2024-02-11T22:42:16.718 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-11T22:42:16.718 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954 --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} --verbose --name nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi --owner scheduled_nmordech@teuthology -- /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954/orig.config.yaml 2024-02-11T22:42:16.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-11T22:42:16.822 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954 branch: wip-nitzan-max-pglog-size description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7539954' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-nitzan-max-pglog-size ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 workunit: branch: wip-nitzan-max-pglog-size sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 owner: scheduled_nmordech@teuthology priority: 1000 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8695 sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-nitzan-max-pglog-size suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhN9lKvvBNcZlCqGC20PdPNscr2lTLGWI4jiQomAfpzmgtfxw8ktVt0xlWefAIztg5Jn4xzzcfICbng9+KMtzE= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-01-31_07:15:30 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-11T22:42:16.822 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa; will attempt to use it 2024-02-11T22:42:16.823 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks 2024-02-11T22:42:16.823 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-11T22:42:16.825 INFO:teuthology.task.internal:Checking packages... 2024-02-11T22:42:16.855 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98' 2024-02-11T22:42:16.855 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-11T22:42:16.855 INFO:teuthology.packaging:ref: None 2024-02-11T22:42:16.855 INFO:teuthology.packaging:tag: None 2024-02-11T22:42:16.855 INFO:teuthology.packaging:branch: wip-nitzan-max-pglog-size 2024-02-11T22:42:16.855 INFO:teuthology.packaging:sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-11T22:42:16.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-nitzan-max-pglog-size 2024-02-11T22:42:17.077 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-658-gd4ed20cf-1jammy 2024-02-11T22:42:17.079 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-11T22:42:17.085 INFO:teuthology.task.internal:no buildpackages task found 2024-02-11T22:42:17.085 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-11T22:42:17.090 INFO:teuthology.task.internal:Saving configuration 2024-02-11T22:42:17.101 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-11T22:42:17.107 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-11T22:42:17.134 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 22:36:38.515468', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN68T2602J+5ABNwkvh4XCOmq3NsAxkC5lrtnfAfPUFw'} 2024-02-11T22:42:17.134 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-11T22:42:17.140 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-11T22:42:17.140 INFO:teuthology.run_tasks:Running task console_log... 2024-02-11T22:42:17.185 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f32203cf160>, signals=[15]) 2024-02-11T22:42:17.186 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-11T22:42:17.193 INFO:teuthology.task.internal:Opening connections... 2024-02-11T22:42:17.193 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-02-11T22:42:17.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:42:17.273 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-11T22:42:17.280 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-02-11T22:42:17.364 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-02-11T22:42:17.365 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-02-11T22:42:17.413 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-11T22:42:17.413 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2024-02-11T22:42:17.413 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="22.04" 2024-02-11T22:42:17.413 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-11T22:42:17.413 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=jammy 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-11T22:42:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=jammy 2024-02-11T22:42:17.415 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-02-11T22:42:17.436 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-11T22:42:17.445 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-11T22:42:17.451 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-11T22:42:17.451 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-02-11T22:42:17.460 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-11T22:42:17.466 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-11T22:42:17.467 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-02-11T22:42:17.511 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-11T22:42:17.543 INFO:teuthology.run_tasks:Running task kernel... 2024-02-11T22:42:17.557 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-11T22:42:17.558 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-11T22:42:17.558 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-11T22:42:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T22:42:17.563 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-11T22:42:17.610 INFO:teuthology.orchestra.run.smithi121.stdout:5.15.0-56-generic 2024-02-11T22:42:17.610 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.15.0-56-generic 2024-02-11T22:42:17.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2024-02-11T22:42:17.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-11T22:42:18.003 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-11T22:42:18.003 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-11T22:42:18.055 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-11T22:42:18.418 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-11T22:42:18.421 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [400 kB] 2024-02-11T22:42:18.554 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [567 kB] 2024-02-11T22:42:18.699 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,142 kB] 2024-02-11T22:42:18.744 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,366 kB] 2024-02-11T22:42:18.859 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [211 kB] 2024-02-11T22:42:18.874 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-11T22:42:18.874 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,366 kB] 2024-02-11T22:42:18.885 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [272 kB] 2024-02-11T22:42:18.900 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-11T22:42:18.901 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-11T22:42:18.903 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,412 kB] 2024-02-11T22:42:18.951 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-11T22:42:18.990 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [224 kB] 2024-02-11T22:42:18.990 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [233 kB] 2024-02-11T22:42:18.994 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-11T22:42:18.995 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-11T22:42:18.995 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [839 kB] 2024-02-11T22:42:19.009 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [590 kB] 2024-02-11T22:42:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,043 kB] 2024-02-11T22:42:19.183 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB] 2024-02-11T22:42:19.578 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-11T22:42:19.584 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-11T22:42:19.598 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-11T22:42:19.599 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-11T22:42:19.601 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-11T22:42:19.782 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [687 kB] 2024-02-11T22:42:19.821 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB] 2024-02-11T22:42:19.835 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-11T22:42:19.835 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,164 B] 2024-02-11T22:42:19.836 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-11T22:42:19.837 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-11T22:42:19.838 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-11T22:42:19.839 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-11T22:42:19.840 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-11T22:42:19.864 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-11T22:42:20.029 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-11T22:42:20.112 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-02-11T22:42:20.208 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-11T22:42:20.236 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-11T22:42:20.288 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-11T22:42:24.554 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 11.5 MB in 2s (4,589 kB/s) 2024-02-11T22:42:25.876 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:42:25.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-11T22:42:25.952 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:42:26.143 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-11T22:42:26.143 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-11T22:42:26.311 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2024-02-11T22:42:26.311 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-11T22:42:26.311 INFO:teuthology.orchestra.run.smithi121.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-11T22:42:26.312 INFO:teuthology.orchestra.run.smithi121.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-11T22:42:26.312 INFO:teuthology.orchestra.run.smithi121.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-11T22:42:26.312 INFO:teuthology.orchestra.run.smithi121.stdout: python2.7-minimal 2024-02-11T22:42:26.312 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-11T22:42:26.313 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2024-02-11T22:42:26.313 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-5.15.0-94 linux-headers-5.15.0-94-generic 2024-02-11T22:42:26.313 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-generic linux-image-5.15.0-94-generic 2024-02-11T22:42:26.313 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-5.15.0-94-generic linux-modules-extra-5.15.0-94-generic 2024-02-11T22:42:26.314 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2024-02-11T22:42:26.315 INFO:teuthology.orchestra.run.smithi121.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-11T22:42:26.315 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2024-02-11T22:42:26.315 INFO:teuthology.orchestra.run.smithi121.stdout: thermald 2024-02-11T22:42:26.366 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2024-02-11T22:42:26.366 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-5.15.0-94 linux-headers-5.15.0-94-generic 2024-02-11T22:42:26.366 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-5.15.0-94-generic linux-modules-5.15.0-94-generic 2024-02-11T22:42:26.367 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-5.15.0-94-generic 2024-02-11T22:42:26.368 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2024-02-11T22:42:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-11T22:42:26.445 INFO:teuthology.orchestra.run.smithi121.stdout:3 upgraded, 5 newly installed, 0 to remove and 282 not upgraded. 2024-02-11T22:42:26.445 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 113 MB of archives. 2024-02-11T22:42:26.445 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-11T22:42:26.445 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-94-generic amd64 5.15.0-94.104 [22.6 MB] 2024-02-11T22:42:26.903 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-94-generic amd64 5.15.0-94.104 [11.5 MB] 2024-02-11T22:42:27.036 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-94-generic amd64 5.15.0-94.104 [63.9 MB] 2024-02-11T22:42:28.167 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.94.91 [1,698 B] 2024-02-11T22:42:28.167 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.94.91 [2,458 B] 2024-02-11T22:42:28.168 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-94 all 5.15.0-94.104 [12.4 MB] 2024-02-11T22:42:28.846 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-94-generic amd64 5.15.0-94.104 [2,877 kB] 2024-02-11T22:42:28.963 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.94.91 [2,306 B] 2024-02-11T22:42:29.362 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 113 MB in 3s (43.7 MB/s) 2024-02-11T22:42:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-5.15.0-94-generic. 2024-02-11T22:42:31.673 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-11T22:42:31.677 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../0-linux-modules-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-11T22:42:31.914 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:33.907 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-5.15.0-94-generic. 2024-02-11T22:42:33.928 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../1-linux-image-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-11T22:42:34.010 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:34.493 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-94-generic. 2024-02-11T22:42:34.514 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-11T22:42:34.551 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:41.277 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../3-linux-generic_5.15.0.94.91_amd64.deb ... 2024-02-11T22:42:41.404 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-11T22:42:41.703 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.94.91_amd64.deb ... 2024-02-11T22:42:41.824 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-11T22:42:42.119 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.15.0-94. 2024-02-11T22:42:42.135 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../5-linux-headers-5.15.0-94_5.15.0-94.104_all.deb ... 2024-02-11T22:42:42.177 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.15.0-94 (5.15.0-94.104) ... 2024-02-11T22:42:48.291 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.15.0-94-generic. 2024-02-11T22:42:48.327 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../6-linux-headers-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-11T22:42:48.366 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:50.970 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.94.91_amd64.deb ... 2024-02-11T22:42:51.093 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-11T22:42:51.434 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.15.0-94 (5.15.0-94.104) ... 2024-02-11T22:42:51.556 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:51.682 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-generic (5.15.0.94.91) ... 2024-02-11T22:42:51.811 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:53.207 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:54.560 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-11T22:42:54.561 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-11T22:42:54.561 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-94-generic 2024-02-11T22:42:54.561 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-94-generic 2024-02-11T22:42:54.691 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:56.068 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic (5.15.0.94.91) ... 2024-02-11T22:42:56.202 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-generic (5.15.0.94.91) ... 2024-02-11T22:42:56.303 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-11T22:42:56.359 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-11T22:42:56.359 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-94-generic 2024-02-11T22:43:10.455 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-11T22:43:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2024-02-11T22:43:10.473 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-11T22:43:10.505 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2024-02-11T22:43:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.15.0-94-generic 2024-02-11T22:43:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.15.0-94-generic 2024-02-11T22:43:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-11T22:43:11.061 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-11T22:43:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-11T22:43:11.142 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-11T22:43:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-11T22:43:11.302 INFO:teuthology.orchestra.run.smithi121.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-11T22:43:11.302 INFO:teuthology.orchestra.run.smithi121.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-11T22:43:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:done 2024-02-11T22:43:11.799 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2024-02-11T22:43:11.823 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 21 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.15.0.94.91 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-94), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.15.0-94-generic, linux-modules-extra-5.15.0-94-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2024-02-11T22:43:11.824 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2024-02-11T22:43:11.825 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2024-02-11T22:43:11.825 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-94-generic 2024-02-11T22:43:11.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-11T22:43:11.825 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-11T22:43:11.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-11T22:43:11.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-02-11T22:43:11.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2024-02-11T22:43:12.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-11T22:43:12.147 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-11T22:43:12.261 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-11T22:43:12.342 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-11T22:43:12.424 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-11T22:43:13.540 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:43:13.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-11T22:43:13.617 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:43:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-11T22:43:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-11T22:43:13.988 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic is already the newest version (5.15.0.94.91). 2024-02-11T22:43:13.988 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2024-02-11T22:43:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-11T22:43:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-11T22:43:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-11T22:43:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-11T22:43:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-11T22:43:13.989 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-11T22:43:14.028 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 282 not upgraded. 2024-02-11T22:43:14.029 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2024-02-11T22:43:14.042 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2024-02-11T22:43:14.042 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2024-02-11T22:43:14.042 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2024-02-11T22:43:14.042 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2024-02-11T22:43:14.042 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 21 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.15.0.94.91 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-94), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.15.0-94-generic, linux-modules-extra-5.15.0-94-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2024-02-11T22:43:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2024-02-11T22:43:14.043 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-02-11T22:43:14.089 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.vNqOAzNp6B 2024-02-11T22:43:14.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vNqOAzNp6B 2024-02-11T22:43:14.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.vNqOAzNp6B 2024-02-11T22:43:14.267 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.vNqOAzNp6B is 10KB 2024-02-11T22:43:14.280 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.vNqOAzNp6B 2024-02-11T22:43:14.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-11T22:43:14.336 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:43:14.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-11T22:43:14.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-11T22:43:14.404 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-94-generic 2024-02-11T22:43:14.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2024-02-11T22:43:15.526 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2024-02-11T22:43:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-11T22:43:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-02-11T22:43:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.15.0-94-generic 2024-02-11T22:43:15.721 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.15.0-94-generic 2024-02-11T22:43:15.956 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-11T22:43:15.962 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-11T22:43:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-11T22:43:16.011 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-11T22:43:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-11T22:43:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-11T22:43:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-11T22:43:16.186 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-02-11T22:43:16.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-02-11T22:43:46.226 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-11T22:43:46.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:43:46.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:44:04.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-11T22:44:13.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:44:13.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:44:16.864 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-11T22:44:28.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:44:28.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:44:47.293 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-11T22:45:02.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:45:02.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:45:02.429 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-11T22:45:02.914 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:45:02.914 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-94-generic"... 2024-02-11T22:45:02.914 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-11T22:45:02.962 INFO:teuthology.orchestra.run.smithi121.stdout:5.15.0-94-generic 2024-02-11T22:45:02.962 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-94-generic vs 5.15.0-94-generic 2024-02-11T22:45:02.962 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-11T22:45:02.962 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-11T22:45:03.962 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-11T22:45:03.963 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-11T22:45:04.018 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-02-11T22:45:04.048 DEBUG:teuthology.parallel:result is None 2024-02-11T22:45:04.049 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-11T22:45:04.057 INFO:teuthology.task.internal:Creating test directory... 2024-02-11T22:45:04.058 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-11T22:45:04.066 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-11T22:45:04.073 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-11T22:45:04.081 INFO:teuthology.task.internal:Creating archive directory... 2024-02-11T22:45:04.081 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-11T22:45:04.126 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-11T22:45:04.132 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-11T22:45:04.133 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-11T22:45:04.197 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:45:04.205 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:45:04.207 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-11T22:45:04.214 INFO:teuthology.task.internal:Configuring sudo... 2024-02-11T22:45:04.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-11T22:45:04.263 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-11T22:45:04.273 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-11T22:45:04.274 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-11T22:45:04.310 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-11T22:45:04.358 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-11T22:45:04.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:45:04.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-11T22:45:04.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-02-11T22:45:04.555 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-11T22:45:04.563 INFO:teuthology.task.internal:Starting timer... 2024-02-11T22:45:04.563 INFO:teuthology.run_tasks:Running task pcp... 2024-02-11T22:45:04.573 INFO:teuthology.run_tasks:Running task selinux... 2024-02-11T22:45:04.584 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2024-02-11T22:45:04.585 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-11T22:45:04.585 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-11T22:45:04.585 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-11T22:45:04.585 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-11T22:45:04.595 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-11T22:45:04.614 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-11T22:45:04.615 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-11T22:52:00.337 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-02-11T22:52:00.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:52:00.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T22:52:00.423 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-11T22:52:00.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-11T22:52:00.505 INFO:teuthology.run_tasks:Running task clock... 2024-02-11T22:52:00.515 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-11T22:52:00.516 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-11T22:52:00.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T22:52:00.572 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-11T22:52:00.572 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Command line: ntpd -gq 2024-02-11T22:52:00.572 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: ---------------------------------------------------- 2024-02-11T22:52:00.572 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: ntp-4 is maintained by Network Time Foundation, 2024-02-11T22:52:00.572 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: corporation. Support and training for ntp-4 are 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: available at https://www.nwtime.org/support 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: ---------------------------------------------------- 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: proto: precision = 0.041 usec (-24) 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: basedate set to 2022-02-04 2024-02-11T22:52:00.573 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: gps base set to 2022-02-06 (week 2196) 2024-02-11T22:52:00.574 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-11T22:52:00.574 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-11T22:52:00.574 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: restrict ::: KOD does nothing without LIMITED. 2024-02-11T22:52:00.574 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-11T22:52:00.575 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen and drop on 0 v6wildcard [::]:123 2024-02-11T22:52:00.575 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-11T22:52:00.575 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen normally on 2 lo 127.0.0.1:123 2024-02-11T22:52:00.576 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2024-02-11T22:52:00.576 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen normally on 4 lo [::1]:123 2024-02-11T22:52:00.576 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2024-02-11T22:52:00.576 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:00 ntpd[17147]: Listening on routing socket on fd #22 for interface updates 2024-02-11T22:52:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:02 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:03.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:03 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:04.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:04.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:04.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:04.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:04.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:04 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:05.572 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:05.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:05.577 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:05.577 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:05.577 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:05.577 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:05 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:06.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:06.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:06.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:06 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:07.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:07.575 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:07.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:07.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:07.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:07.576 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:07 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: can't open /var/log/ntpstats/rawstats.20240211: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-11T22:52:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: can't open /var/log/ntpstats/peerstats.20240211: Permission denied 2024-02-11T22:52:08.574 INFO:teuthology.orchestra.run.smithi121.stdout:11 Feb 22:52:08 ntpd[17147]: ntpd: time slew -0.000542 s 2024-02-11T22:52:08.574 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew -0.000542s 2024-02-11T22:52:08.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-11T22:52:08.574 INFO:teuthology.orchestra.run.smithi121.stderr:11 Feb 22:52:08 ntpd[17147]: can't open /var/log/ntpstats/loopstats.20240211: Permission denied 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-11T22:52:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-11T22:52:08.619 INFO:teuthology.run_tasks:Running task install... 2024-02-11T22:52:08.631 DEBUG:teuthology.task.install:project ceph 2024-02-11T22:52:08.632 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}} 2024-02-11T22:52:08.632 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'} 2024-02-11T22:52:08.632 INFO:teuthology.task.install:Using flavor: default 2024-02-11T22:52:08.638 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-11T22:52:08.638 INFO:teuthology.task.install:extra packages: [] 2024-02-11T22:52:08.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-key list | grep Ceph 2024-02-11T22:52:08.736 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-11T22:52:08.765 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-11T22:52:08.765 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph.com (release key) 2024-02-11T22:52:08.765 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-11T22:52:08.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-11T22:52:08.990 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default/ 2024-02-11T22:52:08.991 INFO:teuthology.task.install.deb:Package version is 19.0.0-658-gd4ed20cf-1jammy 2024-02-11T22:52:09.163 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:52:09.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-11T22:52:09.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-11T22:52:09.353 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-11T22:52:09.353 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-11T22:52:09.366 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-11T22:52:09.386 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-11T22:52:09.460 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy InRelease 2024-02-11T22:52:09.497 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-02-11T22:52:09.533 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-11T22:52:09.570 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-02-11T22:52:09.612 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-02-11T22:52:09.692 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 28.8 kB in 0s (85.3 kB/s) 2024-02-11T22:52:10.609 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:52:10.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-658-gd4ed20cf-1jammy cephadm=19.0.0-658-gd4ed20cf-1jammy ceph-mds=19.0.0-658-gd4ed20cf-1jammy ceph-mgr=19.0.0-658-gd4ed20cf-1jammy ceph-common=19.0.0-658-gd4ed20cf-1jammy ceph-fuse=19.0.0-658-gd4ed20cf-1jammy ceph-test=19.0.0-658-gd4ed20cf-1jammy ceph-volume=19.0.0-658-gd4ed20cf-1jammy radosgw=19.0.0-658-gd4ed20cf-1jammy python3-rados=19.0.0-658-gd4ed20cf-1jammy python3-rgw=19.0.0-658-gd4ed20cf-1jammy python3-cephfs=19.0.0-658-gd4ed20cf-1jammy python3-rbd=19.0.0-658-gd4ed20cf-1jammy libcephfs2=19.0.0-658-gd4ed20cf-1jammy libcephfs-dev=19.0.0-658-gd4ed20cf-1jammy librados2=19.0.0-658-gd4ed20cf-1jammy librbd1=19.0.0-658-gd4ed20cf-1jammy rbd-fuse=19.0.0-658-gd4ed20cf-1jammy 2024-02-11T22:52:10.674 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-11T22:52:10.849 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-11T22:52:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-11T22:52:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2024-02-11T22:52:11.009 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-11T22:52:11.009 INFO:teuthology.orchestra.run.smithi121.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-11T22:52:11.009 INFO:teuthology.orchestra.run.smithi121.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-11T22:52:11.009 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-11T22:52:11.010 INFO:teuthology.orchestra.run.smithi121.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-11T22:52:11.011 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-11T22:52:11.012 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet zip 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-11T22:52:11.013 INFO:teuthology.orchestra.run.smithi121.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2024-02-11T22:52:11.014 INFO:teuthology.orchestra.run.smithi121.stdout: btrfs-tools 2024-02-11T22:52:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2024-02-11T22:52:11.112 INFO:teuthology.orchestra.run.smithi121.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-11T22:52:11.112 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-11T22:52:11.112 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-11T22:52:11.112 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-11T22:52:11.113 INFO:teuthology.orchestra.run.smithi121.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-11T22:52:11.113 INFO:teuthology.orchestra.run.smithi121.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-11T22:52:11.114 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-11T22:52:11.115 INFO:teuthology.orchestra.run.smithi121.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-11T22:52:11.178 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 118 newly installed, 0 to remove and 269 not upgraded. 2024-02-11T22:52:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 172 MB of archives. 2024-02-11T22:52:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 713 MB of additional disk space will be used. 2024-02-11T22:52:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-11T22:52:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-11T22:52:11.283 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-11T22:52:11.303 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-11T22:52:11.333 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-658-gd4ed20cf-1jammy [3,530 kB] 2024-02-11T22:52:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-11T22:52:11.376 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-11T22:52:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-11T22:52:11.392 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-11T22:52:11.392 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-11T22:52:11.393 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-11T22:52:11.401 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-11T22:52:11.403 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-11T22:52:11.405 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-11T22:52:11.407 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-11T22:52:11.415 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-11T22:52:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-02-11T22:52:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-11T22:52:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-11T22:52:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-11T22:52:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-11T22:52:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-11T22:52:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-11T22:52:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-11T22:52:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-11T22:52:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-11T22:52:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-11T22:52:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-11T22:52:11.458 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-11T22:52:11.474 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-11T22:52:11.474 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-11T22:52:11.475 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-11T22:52:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-11T22:52:11.477 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-11T22:52:11.477 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-11T22:52:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-11T22:52:11.494 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-11T22:52:11.495 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-11T22:52:11.500 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-11T22:52:11.513 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-11T22:52:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-11T22:52:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-11T22:52:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-11T22:52:11.520 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-11T22:52:11.521 INFO:teuthology.orchestra.run.smithi121.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-11T22:52:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-11T22:52:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-11T22:52:11.536 INFO:teuthology.orchestra.run.smithi121.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-11T22:52:11.536 INFO:teuthology.orchestra.run.smithi121.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-11T22:52:11.552 INFO:teuthology.orchestra.run.smithi121.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-11T22:52:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-11T22:52:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-11T22:52:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-11T22:52:11.612 INFO:teuthology.orchestra.run.smithi121.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-11T22:52:11.612 INFO:teuthology.orchestra.run.smithi121.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-11T22:52:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-11T22:52:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-11T22:52:11.634 INFO:teuthology.orchestra.run.smithi121.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-11T22:52:11.634 INFO:teuthology.orchestra.run.smithi121.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-11T22:52:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-11T22:52:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-11T22:52:11.654 INFO:teuthology.orchestra.run.smithi121.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-11T22:52:11.654 INFO:teuthology.orchestra.run.smithi121.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-11T22:52:11.655 INFO:teuthology.orchestra.run.smithi121.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-11T22:52:11.655 INFO:teuthology.orchestra.run.smithi121.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-11T22:52:11.656 INFO:teuthology.orchestra.run.smithi121.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-11T22:52:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-11T22:52:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-11T22:52:11.679 INFO:teuthology.orchestra.run.smithi121.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-11T22:52:11.680 INFO:teuthology.orchestra.run.smithi121.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-11T22:52:11.680 INFO:teuthology.orchestra.run.smithi121.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-11T22:52:11.693 INFO:teuthology.orchestra.run.smithi121.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-11T22:52:11.697 INFO:teuthology.orchestra.run.smithi121.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-11T22:52:11.698 INFO:teuthology.orchestra.run.smithi121.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-11T22:52:11.698 INFO:teuthology.orchestra.run.smithi121.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-11T22:52:11.703 INFO:teuthology.orchestra.run.smithi121.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-11T22:52:11.713 INFO:teuthology.orchestra.run.smithi121.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-11T22:52:11.716 INFO:teuthology.orchestra.run.smithi121.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-11T22:52:11.716 INFO:teuthology.orchestra.run.smithi121.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-11T22:52:11.717 INFO:teuthology.orchestra.run.smithi121.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-11T22:52:11.717 INFO:teuthology.orchestra.run.smithi121.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-11T22:52:11.733 INFO:teuthology.orchestra.run.smithi121.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-11T22:52:11.733 INFO:teuthology.orchestra.run.smithi121.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-11T22:52:11.740 INFO:teuthology.orchestra.run.smithi121.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-11T22:52:11.740 INFO:teuthology.orchestra.run.smithi121.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-11T22:52:11.740 INFO:teuthology.orchestra.run.smithi121.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-11T22:52:11.752 INFO:teuthology.orchestra.run.smithi121.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-11T22:52:11.753 INFO:teuthology.orchestra.run.smithi121.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-11T22:52:11.896 INFO:teuthology.orchestra.run.smithi121.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-658-gd4ed20cf-1jammy [3,207 kB] 2024-02-11T22:52:12.152 INFO:teuthology.orchestra.run.smithi121.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-658-gd4ed20cf-1jammy [709 kB] 2024-02-11T22:52:12.192 INFO:teuthology.orchestra.run.smithi121.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-658-gd4ed20cf-1jammy [356 kB] 2024-02-11T22:52:12.204 INFO:teuthology.orchestra.run.smithi121.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-658-gd4ed20cf-1jammy [32.8 kB] 2024-02-11T22:52:12.221 INFO:teuthology.orchestra.run.smithi121.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-658-gd4ed20cf-1jammy [180 kB] 2024-02-11T22:52:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-658-gd4ed20cf-1jammy [64.1 kB] 2024-02-11T22:52:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-658-gd4ed20cf-1jammy [330 kB] 2024-02-11T22:52:12.240 INFO:teuthology.orchestra.run.smithi121.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-658-gd4ed20cf-1jammy [6,225 kB] 2024-02-11T22:52:12.550 INFO:teuthology.orchestra.run.smithi121.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-658-gd4ed20cf-1jammy [112 kB] 2024-02-11T22:52:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-658-gd4ed20cf-1jammy [465 kB] 2024-02-11T22:52:12.568 INFO:teuthology.orchestra.run.smithi121.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-658-gd4ed20cf-1jammy [23.0 MB] 2024-02-11T22:52:13.411 INFO:teuthology.orchestra.run.smithi121.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-658-gd4ed20cf-1jammy [5,057 kB] 2024-02-11T22:52:13.499 INFO:teuthology.orchestra.run.smithi121.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-658-gd4ed20cf-1jammy [240 kB] 2024-02-11T22:52:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-658-gd4ed20cf-1jammy [124 kB] 2024-02-11T22:52:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-658-gd4ed20cf-1jammy [1,519 kB] 2024-02-11T22:52:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-658-gd4ed20cf-1jammy [6,238 kB] 2024-02-11T22:52:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-658-gd4ed20cf-1jammy [22.7 MB] 2024-02-11T22:52:14.639 INFO:teuthology.orchestra.run.smithi121.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-658-gd4ed20cf-1jammy [14.1 kB] 2024-02-11T22:52:14.640 INFO:teuthology.orchestra.run.smithi121.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-658-gd4ed20cf-1jammy [866 kB] 2024-02-11T22:52:14.657 INFO:teuthology.orchestra.run.smithi121.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-658-gd4ed20cf-1jammy [2,182 kB] 2024-02-11T22:52:14.696 INFO:teuthology.orchestra.run.smithi121.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-658-gd4ed20cf-1jammy [770 kB] 2024-02-11T22:52:14.705 INFO:teuthology.orchestra.run.smithi121.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-658-gd4ed20cf-1jammy [139 kB] 2024-02-11T22:52:14.707 INFO:teuthology.orchestra.run.smithi121.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-658-gd4ed20cf-1jammy [1,876 kB] 2024-02-11T22:52:14.742 INFO:teuthology.orchestra.run.smithi121.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-658-gd4ed20cf-1jammy [8,625 kB] 2024-02-11T22:52:15.027 INFO:teuthology.orchestra.run.smithi121.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-658-gd4ed20cf-1jammy [14.2 kB] 2024-02-11T22:52:15.027 INFO:teuthology.orchestra.run.smithi121.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-658-gd4ed20cf-1jammy [48.4 MB] 2024-02-11T22:52:16.794 INFO:teuthology.orchestra.run.smithi121.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-658-gd4ed20cf-1jammy [121 kB] 2024-02-11T22:52:16.873 INFO:teuthology.orchestra.run.smithi121.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-658-gd4ed20cf-1jammy [38.4 kB] 2024-02-11T22:52:16.874 INFO:teuthology.orchestra.run.smithi121.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-658-gd4ed20cf-1jammy [12.3 MB] 2024-02-11T22:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-658-gd4ed20cf-1jammy [91.7 kB] 2024-02-11T22:52:17.769 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 172 MB in 6s (28.2 MB/s) 2024-02-11T22:52:17.935 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-11T22:52:18.034 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165994 files and directories currently installed.) 2024-02-11T22:52:18.040 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-11T22:52:18.125 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-11T22:52:18.470 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-11T22:52:18.497 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-11T22:52:18.537 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-11T22:52:18.932 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-11T22:52:18.958 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-11T22:52:19.007 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-11T22:52:19.469 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-11T22:52:19.496 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-11T22:52:19.543 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:52:19.922 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-11T22:52:19.948 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-11T22:52:19.988 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:52:20.333 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-11T22:52:20.359 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-11T22:52:20.400 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:52:20.745 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-11T22:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-11T22:52:20.811 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-11T22:52:21.089 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librados2. 2024-02-11T22:52:21.116 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../007-librados2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:21.156 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librados2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:21.609 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librbd1. 2024-02-11T22:52:21.635 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../008-librbd1_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:21.675 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librbd1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:22.111 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs2. 2024-02-11T22:52:22.138 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:22.178 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:22.431 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rados. 2024-02-11T22:52:22.457 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../010-python3-rados_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:22.489 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rados (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:22.825 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-11T22:52:22.851 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:22.891 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-argparse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:23.152 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cephfs. 2024-02-11T22:52:23.178 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:23.219 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cephfs (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:23.530 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-11T22:52:23.556 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:23.597 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:23.908 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-11T22:52:23.935 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-11T22:52:23.975 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-11T22:52:24.295 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-prettytable. 2024-02-11T22:52:24.321 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-11T22:52:24.361 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-11T22:52:24.689 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rbd. 2024-02-11T22:52:24.715 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:24.755 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rbd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:25.058 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-11T22:52:25.084 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-11T22:52:25.116 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-11T22:52:25.394 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-11T22:52:25.421 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-11T22:52:25.461 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-11T22:52:25.764 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-11T22:52:25.779 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-11T22:52:25.822 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-11T22:52:26.142 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package lua5.1. 2024-02-11T22:52:26.169 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-11T22:52:26.208 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-11T22:52:26.512 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package lua-any. 2024-02-11T22:52:26.528 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-11T22:52:26.570 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-11T22:52:26.823 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package zip. 2024-02-11T22:52:26.849 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-11T22:52:26.889 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking zip (3.0-12build2) ... 2024-02-11T22:52:27.201 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package unzip. 2024-02-11T22:52:27.227 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-02-11T22:52:27.268 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-02-11T22:52:27.596 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package luarocks. 2024-02-11T22:52:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-11T22:52:27.663 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-11T22:52:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librgw2. 2024-02-11T22:52:28.048 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../025-librgw2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librgw2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:28.640 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rgw. 2024-02-11T22:52:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:29.001 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-11T22:52:29.027 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-11T22:52:29.059 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-11T22:52:29.462 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libradosstriper1. 2024-02-11T22:52:29.489 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libradosstriper1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-common. 2024-02-11T22:52:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../029-ceph-common_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:31.246 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-base. 2024-02-11T22:52:31.272 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../030-ceph-base_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:31.322 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-base (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:31.875 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-11T22:52:31.901 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-11T22:52:31.941 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-11T22:52:32.278 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cheroot. 2024-02-11T22:52:32.304 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-11T22:52:32.344 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-11T22:52:32.664 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-11T22:52:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-11T22:52:32.730 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-11T22:52:33.008 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-11T22:52:33.035 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-11T22:52:33.075 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-11T22:52:33.372 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-11T22:52:33.398 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-11T22:52:33.438 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-11T22:52:33.741 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-tempora. 2024-02-11T22:52:33.767 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-11T22:52:33.807 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-11T22:52:34.327 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-portend. 2024-02-11T22:52:34.356 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-11T22:52:34.385 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-11T22:52:34.680 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-11T22:52:34.706 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-11T22:52:34.746 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-11T22:52:35.041 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-11T22:52:35.067 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-11T22:52:35.107 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-11T22:52:35.463 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-natsort. 2024-02-11T22:52:35.487 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-11T22:52:35.527 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-11T22:52:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-logutils. 2024-02-11T22:52:35.865 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-11T22:52:35.905 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-11T22:52:36.200 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-11T22:52:36.226 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-11T22:52:36.266 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-11T22:52:36.519 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-mako. 2024-02-11T22:52:36.546 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-11T22:52:36.586 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-11T22:52:36.914 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-11T22:52:36.940 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-11T22:52:36.980 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-11T22:52:37.283 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-11T22:52:37.309 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-11T22:52:37.350 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-11T22:52:37.636 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webob. 2024-02-11T22:52:37.662 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-11T22:52:37.702 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-11T22:52:37.980 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-waitress. 2024-02-11T22:52:38.006 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-11T22:52:38.041 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-11T22:52:38.325 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-tempita. 2024-02-11T22:52:38.351 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-11T22:52:38.391 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-11T22:52:38.694 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-paste. 2024-02-11T22:52:38.720 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-11T22:52:38.760 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-11T22:52:39.039 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-11T22:52:39.054 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-11T22:52:39.097 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-11T22:52:39.425 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-11T22:52:39.451 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-11T22:52:39.491 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-11T22:52:39.794 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webtest. 2024-02-11T22:52:39.821 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-11T22:52:39.861 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-11T22:52:40.180 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pecan. 2024-02-11T22:52:40.207 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-11T22:52:40.247 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-11T22:52:40.618 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-certifi. 2024-02-11T22:52:40.645 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-11T22:52:40.685 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-11T22:52:40.996 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-idna. 2024-02-11T22:52:41.022 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-11T22:52:41.062 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-11T22:52:41.382 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-urllib3. 2024-02-11T22:52:41.409 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-11T22:52:41.448 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-11T22:52:41.785 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-requests. 2024-02-11T22:52:41.814 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-11T22:52:41.852 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-11T22:52:42.165 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-11T22:52:42.190 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-11T22:52:42.232 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-11T22:52:42.524 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-11T22:52:42.551 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:42.590 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-modules-core (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:42.946 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-11T22:52:42.972 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:43.012 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:43.306 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-11T22:52:43.333 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-11T22:52:43.373 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-11T22:52:43.676 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-openssl. 2024-02-11T22:52:43.702 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-11T22:52:43.760 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-11T22:52:44.112 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr. 2024-02-11T22:52:44.139 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:44.178 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:44.990 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mon. 2024-02-11T22:52:45.017 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:45.332 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mon (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:45.877 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-osd. 2024-02-11T22:52:45.904 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:45.943 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-osd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:46.988 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph. 2024-02-11T22:52:47.003 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../066-ceph_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:47.046 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:47.374 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-fuse. 2024-02-11T22:52:47.400 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:47.440 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:47.835 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mds. 2024-02-11T22:52:47.861 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:47.902 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mds (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:48.263 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephadm. 2024-02-11T22:52:48.290 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../069-cephadm_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:52:48.321 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:48.641 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-11T22:52:48.668 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-11T22:52:48.707 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-11T22:52:49.018 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-11T22:52:49.045 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-11T22:52:49.084 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-11T22:52:50.257 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-babel. 2024-02-11T22:52:50.284 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-11T22:52:50.322 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-11T22:52:50.642 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jinja2. 2024-02-11T22:52:50.658 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-11T22:52:50.692 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-11T22:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-11T22:52:51.019 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:51.053 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:51.373 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-11T22:52:51.389 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-11T22:52:51.438 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-11T22:52:51.701 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-routes. 2024-02-11T22:52:51.727 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-11T22:52:51.767 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-11T22:52:52.071 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-11T22:52:52.097 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-dashboard (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:52.574 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-11T22:52:52.600 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-11T22:52:52.640 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-11T22:52:53.135 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-joblib. 2024-02-11T22:52:53.162 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-11T22:52:53.201 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-11T22:52:53.563 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-11T22:52:53.590 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-11T22:52:53.630 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-11T22:52:53.932 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn. 2024-02-11T22:52:53.959 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-11T22:52:53.999 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-11T22:52:54.691 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-11T22:52:54.719 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:54.842 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:55.888 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cachetools. 2024-02-11T22:52:55.915 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-11T22:52:55.954 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-11T22:52:56.257 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-11T22:52:56.284 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-11T22:52:56.323 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-11T22:52:56.660 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-11T22:52:56.688 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-11T22:52:56.727 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-11T22:52:57.047 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rsa. 2024-02-11T22:52:57.074 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-11T22:52:57.113 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-11T22:52:57.417 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-google-auth. 2024-02-11T22:52:57.444 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-11T22:52:57.483 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-11T22:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-11T22:52:57.822 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-11T22:52:57.853 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-11T22:52:58.139 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-websocket. 2024-02-11T22:52:58.167 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-11T22:52:58.214 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-11T22:52:58.525 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-11T22:52:58.553 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-11T22:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-11T22:52:59.298 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-11T22:52:59.325 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:52:59.364 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:52:59.641 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-11T22:52:59.657 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-11T22:52:59.691 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-11T22:53:00.020 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-11T22:53:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-11T22:53:00.086 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-11T22:53:00.355 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package jq. 2024-02-11T22:53:00.371 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-11T22:53:00.414 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-11T22:53:00.666 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2024-02-11T22:53:00.693 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-11T22:53:00.733 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-11T22:53:01.011 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package xmlstarlet. 2024-02-11T22:53:01.027 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-11T22:53:01.069 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-11T22:53:01.440 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-test. 2024-02-11T22:53:01.456 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../097-ceph-test_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:53:01.498 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-test (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:03.627 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-volume. 2024-02-11T22:53:03.655 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-11T22:53:03.694 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-volume (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:03.997 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-11T22:53:04.014 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:53:04.056 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs-dev (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:04.350 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-11T22:53:04.378 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-11T22:53:04.417 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-11T22:53:04.705 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-11T22:53:04.732 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-11T22:53:04.771 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-11T22:53:05.562 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package nvme-cli. 2024-02-11T22:53:05.589 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-11T22:53:05.633 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-11T22:53:06.039 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package pkg-config. 2024-02-11T22:53:06.066 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-11T22:53:06.096 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-11T22:53:06.324 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-11T22:53:06.352 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-11T22:53:06.391 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-11T22:53:06.819 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-11T22:53:06.847 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-11T22:53:06.885 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-11T22:53:07.172 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastescript. 2024-02-11T22:53:07.199 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-11T22:53:07.238 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-11T22:53:07.558 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pluggy. 2024-02-11T22:53:07.586 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-11T22:53:07.625 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-11T22:53:07.936 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-psutil. 2024-02-11T22:53:07.963 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-11T22:53:08.004 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-11T22:53:08.331 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-py. 2024-02-11T22:53:08.358 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-11T22:53:08.397 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-11T22:53:08.717 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pygments. 2024-02-11T22:53:08.745 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-11T22:53:08.783 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-11T22:53:09.232 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-11T22:53:09.257 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-11T22:53:09.296 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-11T22:53:09.615 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-toml. 2024-02-11T22:53:09.643 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-11T22:53:09.681 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-11T22:53:09.984 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pytest. 2024-02-11T22:53:10.010 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-11T22:53:10.052 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-11T22:53:10.401 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplejson. 2024-02-11T22:53:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-11T22:53:10.464 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-11T22:53:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-11T22:53:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-11T22:53:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-11T22:53:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package radosgw. 2024-02-11T22:53:11.507 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../116-radosgw_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:53:11.537 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking radosgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package rbd-fuse. 2024-02-11T22:53:12.306 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-11T22:53:12.348 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking rbd-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-11T22:53:12.903 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-11T22:53:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-11T22:53:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-11T22:53:13.613 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-11T22:53:14.039 INFO:teuthology.orchestra.run.smithi121.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-11T22:53:14.106 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:14.657 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user cephadm....done 2024-02-11T22:53:14.811 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-11T22:53:15.088 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-11T22:53:15.314 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-11T22:53:15.415 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-11T22:53:15.666 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-11T22:53:15.909 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-11T22:53:16.035 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-11T22:53:16.370 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-11T22:53:16.747 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-11T22:53:16.989 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-11T22:53:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-02-11T22:53:17.484 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-11T22:53:17.735 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-11T22:53:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-11T22:53:18.245 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-argparse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:18.652 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-11T22:53:18.814 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-11T22:53:18.915 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-11T22:53:19.016 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-11T22:53:19.259 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-11T22:53:19.665 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-11T22:53:19.699 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-11T22:53:19.770 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-11T22:53:19.880 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-11T22:53:20.214 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-11T22:53:20.315 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-11T22:53:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-11T22:53:20.801 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-11T22:53:21.269 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up zip (3.0-12build2) ... 2024-02-11T22:53:21.395 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-11T22:53:22.096 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-11T22:53:22.364 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-11T22:53:22.465 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-11T22:53:22.566 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-11T22:53:22.867 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-11T22:53:23.252 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-11T22:53:23.504 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-11T22:53:23.880 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-11T22:53:24.181 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-11T22:53:24.524 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-11T22:53:24.758 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-11T22:53:24.884 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:25.186 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-idna (3.3-1) ... 2024-02-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-11T22:53:26.625 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-11T22:53:26.868 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:53:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-11T22:53:27.262 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-11T22:53:27.372 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-11T22:53:27.473 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-11T22:53:27.749 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-11T22:53:28.000 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-11T22:53:28.243 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:53:28.369 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-11T22:53:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-11T22:53:28.938 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-11T22:53:29.181 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-11T22:53:29.415 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-11T22:53:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-11T22:53:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-11T22:53:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-11T22:53:30.353 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-11T22:53:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-11T22:53:30.789 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-11T22:53:31.098 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-11T22:53:31.349 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-11T22:53:31.475 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-11T22:53:31.601 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-11T22:53:31.691 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-11T22:53:31.927 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-11T22:53:32.045 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-11T22:53:32.413 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-11T22:53:32.656 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-11T22:53:32.757 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-11T22:53:33.008 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-11T22:53:33.134 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-11T22:53:33.452 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-11T22:53:33.761 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-11T22:53:34.038 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-11T22:53:34.139 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-11T22:53:34.390 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-11T22:53:34.666 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-11T22:53:35.011 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-11T22:53:35.404 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-11T22:53:35.697 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-11T22:53:36.016 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-11T22:53:36.142 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librados2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:36.293 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libsqlite3-mod-ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:36.428 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-11T22:53:36.730 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:36.831 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libradosstriper1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:36.941 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-11T22:53:37.217 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librbd1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:37.343 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-modules-core (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:37.469 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:37.698 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-11T22:53:37.698 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-11T22:53:38.179 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs-dev (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:38.305 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rados (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:38.456 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librgw2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:38.607 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-11T22:53:39.852 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rbd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:39.978 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up rbd-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:40.104 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:40.247 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cephfs (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:40.398 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:40.975 INFO:teuthology.orchestra.run.smithi121.stdout:Adding group ceph....done 2024-02-11T22:53:41.393 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user ceph....done 2024-02-11T22:53:41.560 INFO:teuthology.orchestra.run.smithi121.stdout:Setting system user ceph properties....done 2024-02-11T22:53:41.576 INFO:teuthology.orchestra.run.smithi121.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-11T22:53:41.712 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-11T22:53:42.042 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-11T22:53:42.491 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-test (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:42.617 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up radosgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:43.104 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-11T22:53:43.104 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-11T22:53:43.577 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-base (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:43.904 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-11T22:53:44.355 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mds (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:44.584 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-11T22:53:44.585 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-11T22:53:45.072 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:45.326 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-11T22:53:45.326 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-11T22:53:45.807 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-osd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:46.076 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-11T22:53:46.076 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-11T22:53:46.483 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-k8sevents (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:46.609 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:47.045 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mon (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:47.259 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-11T22:53:47.259 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-11T22:53:47.771 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:47.939 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:48.090 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-dashboard (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-volume (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-11T22:53:48.585 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-11T22:53:48.694 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-11T22:53:48.830 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-11T22:53:49.964 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-11T22:53:50.622 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-11T22:53:50.627 DEBUG:teuthology.parallel:result is None 2024-02-11T22:53:50.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-11T22:53:50.847 DEBUG:teuthology.orchestra.run.smithi121:> dpkg-query -W -f '${Version}' ceph 2024-02-11T22:53:50.866 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-658-gd4ed20cf-1jammy 2024-02-11T22:53:50.866 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658-gd4ed20cf-1jammy 2024-02-11T22:53:50.866 INFO:teuthology.task.install:The correct ceph version 19.0.0-658-gd4ed20cf-1jammy is installed. 2024-02-11T22:53:50.868 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-11T22:53:50.868 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:53:50.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-11T22:53:50.922 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-11T22:53:50.922 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:53:50.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-02-11T22:53:50.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-11T22:53:51.029 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-11T22:53:51.029 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:53:51.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-11T22:53:51.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-11T22:53:51.133 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-11T22:53:51.133 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:53:51.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-02-11T22:53:51.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-11T22:53:51.237 INFO:teuthology.run_tasks:Running task workunit... 2024-02-11T22:53:51.255 INFO:tasks.workunit:Pulling workunits from ref d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-11T22:53:51.255 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-11T22:53:51.255 INFO:tasks.workunit:timeout=3h 2024-02-11T22:53:51.255 INFO:tasks.workunit:cleanup=True 2024-02-11T22:53:51.256 DEBUG:teuthology.orchestra.run.smithi121:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-11T22:53:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T22:53:51.282 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-11T22:53:51.282 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-11T22:53:51.330 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-11T22:53:51.330 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-11T22:53:51.379 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-11T22:53:51.428 INFO:tasks.workunit.client.0.smithi121.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-11T22:54:59.376 INFO:tasks.workunit.client.0.smithi121.stderr:Note: switching to 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'. 2024-02-11T22:54:59.376 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.376 INFO:tasks.workunit.client.0.smithi121.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-11T22:54:59.376 INFO:tasks.workunit.client.0.smithi121.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-11T22:54:59.376 INFO:tasks.workunit.client.0.smithi121.stderr:state without impacting any branches by switching back to a branch. 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: git switch -c 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr:Or undo this operation with: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: git switch - 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-11T22:54:59.377 INFO:tasks.workunit.client.0.smithi121.stderr:HEAD is now at d4ed20cfb23 ceph_dencoder: memory leaks when loading dencoder plugins 2024-02-11T22:54:59.382 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-11T22:54:59.431 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-11T22:54:59.431 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-11T22:54:59.478 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-02-11T22:54:59.479 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-02-11T22:54:59.479 DEBUG:teuthology.orchestra.run.smithi121:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-02-11T22:54:59.538 INFO:tasks.workunit.client.0.smithi121.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-11T22:54:59.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/bad-inc-map 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:41469 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:41469 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none ' 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:41469 ' 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-02-11T22:54:59.573 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T22:54:59.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:54:59.576 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:54:59.577 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T22:54:59.577 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T22:54:59.577 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T22:54:59.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T22:54:59.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T22:54:59.578 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T22:54:59.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:54:59.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T22:54:59.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T22:54:59.579 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:54:59.580 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T22:54:59.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-11T22:54:59.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-02-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20211 2024-02-11T22:54:59.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T22:54:59.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T22:54:59.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-02-11T22:54:59.584 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-11T22:54:59.584 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:54:59.584 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:54:59.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20211 2024-02-11T22:54:59.586 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-02-11T22:54:59.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-02-11T22:55:00.076 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-11T22:55:00.076 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:55:00.076 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:55:00.077 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:55:00.077 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:00.077 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:00.077 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-11T22:55:00.297 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-11T22:55:00.298 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-11T22:55:00.298 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T22:55:00.299 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T22:55:00.299 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:00.300 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20211/ceph-mon.a.asok 2024-02-11T22:55:00.301 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T22:55:00.301 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20211/ceph-mon.a.asok config get fsid 2024-02-11T22:55:00.421 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-11T22:55:00.421 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T22:55:00.421 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T22:55:00.421 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20211/ceph-mon.a.asok 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T22:55:00.423 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20211/ceph-mon.a.asok config get mon_host 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-11T22:55:00.532 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-02-11T22:55:00.533 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:01.437 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:01.438 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:01.438 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T22:55:01.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T22:55:01.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-02-11T22:55:01.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-02-11T22:55:01.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 ' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:55:01.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-02-11T22:55:01.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=475fb097-e760-4472-93b4-714d3615e74d 2024-02-11T22:55:01.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 475fb097-e760-4472-93b4-714d3615e74d' 2024-02-11T22:55:01.530 INFO:tasks.workunit.client.0.smithi121.stdout:add osd0 475fb097-e760-4472-93b4-714d3615e74d 2024-02-11T22:55:01.530 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:55:01.546 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFUMll9Sh+IBAAA5POzX/IGaxy1WkKWn34FA== 2024-02-11T22:55:01.546 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFUMll9Sh+IBAAA5POzX/IGaxy1WkKWn34FA=="}' 2024-02-11T22:55:01.546 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 475fb097-e760-4472-93b4-714d3615e74d -i td/bad-inc-map/0/new.json 2024-02-11T22:55:01.843 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:55:01.860 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-02-11T22:55:01.861 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFUMll9Sh+IBAAA5POzX/IGaxy1WkKWn34FA== --osd-uuid 475fb097-e760-4472-93b4-714d3615e74d 2024-02-11T22:55:01.937 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.932+0000 7ff6f656e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:01.939 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.936+0000 7ff6f656e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:01.941 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.936+0000 7ff6f656e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:01.942 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.936+0000 7ff6f656e640 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-02-11T22:55:01.942 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.936+0000 7ff6f656e640 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-02-11T22:55:01.942 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:01.936+0000 7ff6f656e640 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-02-11T22:55:02.037 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:02.032+0000 7ff6f656e640 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-02-11T22:55:08.343 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-02-11T22:55:08.343 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:55:08.344 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd0 key to auth repository 2024-02-11T22:55:08.344 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-11T22:55:08.344 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:55:08.577 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.0 2024-02-11T22:55:08.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-11T22:55:08.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:55:08.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:55:08.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:55:08.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:55:08.606 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:08.604+0000 7f2f4ff9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:08.608 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:08.604+0000 7f2f4ff9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:08.609 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:08.604+0000 7f2f4ff9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:09.012 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:09.165 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:09.160+0000 7f2f4ff9d640 -1 Falling back to public interface 2024-02-11T22:55:10.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:10.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:10.016 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:55:10.017 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:55:10.017 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:10.017 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:10.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:11.290 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:11.290 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:11.291 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:55:11.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:55:11.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:11.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:11.705 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:11.884 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:11.880+0000 7f2f4ff9d640 -1 osd.0 0 log_to_monitors true 2024-02-11T22:55:12.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:12.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:12.707 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:55:12.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:55:12.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:12.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:13.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:14.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:14.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:14.122 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:55:14.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:55:14.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:14.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:14.539 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:15.541 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:15.541 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:15.542 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:55:15.542 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:55:15.542 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:15.542 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:16.020 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:17.022 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:17.022 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:17.023 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:55:17.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:55:17.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:17.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:17.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:18.466 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:18.466 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:18.466 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:55:18.466 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:55:18.467 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:18.467 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:18.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:19.957 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:55:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:19.959 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:20.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:21.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:21.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:21.392 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:55:21.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:55:21.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:21.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:21.901 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:22.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:22.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:22.903 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:55:22.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:55:22.904 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:22.904 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:23.327 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:24.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:24.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:24.329 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:55:24.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:55:24.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:24.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:24.871 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:25.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:25.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:25.873 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:55:25.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:55:25.874 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:25.874 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:26.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:27.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:27.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:27.584 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:55:27.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:55:27.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:27.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:28.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:29.386 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:29.387 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:29.387 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:55:29.387 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:55:29.387 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:29.387 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:30.027 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:31.028 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:31.028 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:31.029 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:55:31.029 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:55:31.029 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:31.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:31.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:32.683 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:32.683 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:32.683 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:55:32.684 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:55:32.684 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:32.684 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:33.630 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:34.631 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:34.631 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:34.632 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:55:34.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:55:34.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:34.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:36.344 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:37.345 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:37.345 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:37.346 INFO:tasks.workunit.client.0.smithi121.stdout:18 2024-02-11T22:55:37.346 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-11T22:55:37.346 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:37.346 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:38.172 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:39.173 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:39.173 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:39.174 INFO:tasks.workunit.client.0.smithi121.stdout:19 2024-02-11T22:55:39.174 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-11T22:55:39.174 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:39.174 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:55:39.585 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1285530072,v1:127.0.0.1:6801/1285530072] [v2:127.0.0.1:6802/1285530072,v1:127.0.0.1:6803/1285530072] exists,up 475fb097-e760-4472-93b4-714d3615e74d 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 ' 2024-02-11T22:55:39.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:55:39.588 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:39.589 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:55:39.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-02-11T22:55:39.617 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:55:39.618 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=003f4496-7202-4368-8681-eef69f484826 2024-02-11T22:55:39.618 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 003f4496-7202-4368-8681-eef69f484826' 2024-02-11T22:55:39.618 INFO:tasks.workunit.client.0.smithi121.stdout:add osd1 003f4496-7202-4368-8681-eef69f484826 2024-02-11T22:55:39.618 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:55:39.633 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrUMlljA+yJRAAzCJQQNQ4fReHuxIsDcfO3A== 2024-02-11T22:55:39.633 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrUMlljA+yJRAAzCJQQNQ4fReHuxIsDcfO3A=="}' 2024-02-11T22:55:39.634 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 003f4496-7202-4368-8681-eef69f484826 -i td/bad-inc-map/1/new.json 2024-02-11T22:55:40.108 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:55:40.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-02-11T22:55:40.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrUMlljA+yJRAAzCJQQNQ4fReHuxIsDcfO3A== --osd-uuid 003f4496-7202-4368-8681-eef69f484826 2024-02-11T22:55:40.154 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.151+0000 7f0c2dd22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:40.217 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.211+0000 7f0c2dd22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:40.218 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.215+0000 7f0c2dd22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:40.219 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.215+0000 7f0c2dd22640 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-02-11T22:55:40.219 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.215+0000 7f0c2dd22640 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-02-11T22:55:40.219 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.215+0000 7f0c2dd22640 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-02-11T22:55:40.341 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:40.335+0000 7f0c2dd22640 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-02-11T22:55:46.476 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-02-11T22:55:46.477 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:55:46.478 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd1 key to auth repository 2024-02-11T22:55:46.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-11T22:55:46.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:55:47.015 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T22:55:47.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-11T22:55:47.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:55:47.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:55:47.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:55:47.017 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:55:47.039 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:47.035+0000 7f4fd454f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:47.041 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:47.035+0000 7f4fd454f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:47.042 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:47.039+0000 7f4fd454f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:55:47.472 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:55:47.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:47.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:55:47.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:47.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:47.588 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:47.583+0000 7f4fd454f640 -1 Falling back to public interface 2024-02-11T22:55:47.879 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:48.881 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:48.881 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:48.882 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:55:48.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:55:48.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:48.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:49.292 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:50.263 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:55:50.259+0000 7f4fd454f640 -1 osd.1 0 log_to_monitors true 2024-02-11T22:55:50.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:50.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:50.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:55:50.294 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:55:50.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:50.295 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:50.853 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:51.854 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:51.855 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:51.855 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:55:51.855 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:55:51.856 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:51.856 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:52.302 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:53.304 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:53.304 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:53.305 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:55:53.305 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:55:53.305 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:53.305 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:53.738 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:54.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:54.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:54.740 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:55:54.741 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:55:54.741 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:54.741 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:55.196 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:56.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:56.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:56.198 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:55:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:55:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:56.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:57.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:57.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:57.677 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:55:57.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:55:57.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:57.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:58.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:55:59.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:55:59.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:55:59.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:55:59.202 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:55:59.203 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:55:59.203 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:55:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:00.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:00.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:00.643 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:56:00.643 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:56:00.643 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:00.644 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:01.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:02.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:02.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:02.293 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:56:02.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:56:02.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:02.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:02.881 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:03.883 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:03.883 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:03.884 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:56:03.885 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:56:03.885 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:03.885 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:04.733 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:05.733 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:05.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:05.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:56:05.734 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:56:05.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:05.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:06.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:07.797 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:07.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:07.798 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:56:07.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:56:07.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:07.798 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:08.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:09.544 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:09.544 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:09.545 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:56:09.545 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:56:09.545 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:09.545 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:10.282 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:11.284 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:11.284 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:11.284 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:56:11.285 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:56:11.285 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:11.285 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:11.946 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:12.947 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:12.948 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:12.948 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:56:12.949 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:56:12.949 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:12.949 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:13.940 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:14.941 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:14.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:14.942 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:56:14.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:56:14.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:14.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:56:15.349 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1248167582,v1:127.0.0.1:6811/1248167582] [v2:127.0.0.1:6812/1248167582,v1:127.0.0.1:6813/1248167582] exists,up 003f4496-7202-4368-8681-eef69f484826 2024-02-11T22:56:15.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:56:15.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:56:15.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:56:15.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 ' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:56:15.353 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-02-11T22:56:15.354 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-02-11T22:56:15.355 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:56:15.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9714de9c-69be-4472-ac9c-f5abdc9becb2 2024-02-11T22:56:15.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9714de9c-69be-4472-ac9c-f5abdc9becb2' 2024-02-11T22:56:15.356 INFO:tasks.workunit.client.0.smithi121.stdout:add osd2 9714de9c-69be-4472-ac9c-f5abdc9becb2 2024-02-11T22:56:15.356 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:56:15.376 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPUMllG35XFhAAqAsFHAQRNyMENVPg4drzCw== 2024-02-11T22:56:15.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPUMllG35XFhAAqAsFHAQRNyMENVPg4drzCw=="}' 2024-02-11T22:56:15.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9714de9c-69be-4472-ac9c-f5abdc9becb2 -i td/bad-inc-map/2/new.json 2024-02-11T22:56:15.919 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:56:15.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-02-11T22:56:15.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPUMllG35XFhAAqAsFHAQRNyMENVPg4drzCw== --osd-uuid 9714de9c-69be-4472-ac9c-f5abdc9becb2 2024-02-11T22:56:15.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:15.962+0000 7f3265100640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:16.005 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:15.998+0000 7f3265100640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:16.006 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:15.998+0000 7f3265100640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:16.007 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:16.002+0000 7f3265100640 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-02-11T22:56:16.007 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:16.002+0000 7f3265100640 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-02-11T22:56:16.007 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:16.002+0000 7f3265100640 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-02-11T22:56:16.044 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:16.038+0000 7f3265100640 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-02-11T22:56:22.684 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-02-11T22:56:22.684 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:56:22.685 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd2 key to auth repository 2024-02-11T22:56:22.685 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-11T22:56:22.685 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:56:23.211 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-11T22:56:23.212 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.2 2024-02-11T22:56:23.212 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7db7d086-b51c-4903-acb8-f5fdc8554584 --auth-supported=none --mon-host=127.0.0.1:41469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20211/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:56:23.212 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:56:23.212 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:56:23.212 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:56:23.242 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:23.234+0000 7fc962234640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:23.244 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:23.238+0000 7fc962234640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:23.245 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:23.238+0000 7fc962234640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:56:23.649 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-11T22:56:23.649 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:56:23.649 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T22:56:23.649 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:23.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:23.821 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:23.814+0000 7fc962234640 -1 Falling back to public interface 2024-02-11T22:56:24.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:25.059 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:25.059 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:25.060 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:56:25.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:56:25.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:25.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:25.462 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:26.456 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:56:26.450+0000 7fc962234640 -1 osd.2 0 log_to_monitors true 2024-02-11T22:56:26.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:26.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:26.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:56:26.464 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:56:26.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:26.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:27.158 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:28.160 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:28.160 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:28.160 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:56:28.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:56:28.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:28.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:28.864 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:29.865 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:29.865 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:29.866 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:56:29.866 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:56:29.866 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:29.866 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:30.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:31.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:31.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:31.529 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:56:31.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:56:31.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:31.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:31.999 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:33.001 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:33.001 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:33.002 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:56:33.002 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:56:33.002 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:33.002 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:33.444 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:34.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:34.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:34.446 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:56:34.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:56:34.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:34.447 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:34.978 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:35.980 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:35.980 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:35.980 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:56:35.980 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:56:35.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:35.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:36.520 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:37.522 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:37.522 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:37.522 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:56:37.523 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:56:37.523 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:37.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:37.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:38.941 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:38.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:38.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:56:38.942 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:56:38.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:38.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:39.427 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:40.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:40.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:40.429 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:56:40.430 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:56:40.430 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:40.430 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:41.387 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:42.388 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:42.389 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:42.389 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:56:42.389 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:56:42.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:42.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:43.269 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:44.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:44.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:44.271 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:56:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:56:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:45.173 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:46.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:46.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:46.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:56:46.175 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:56:46.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:46.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:47.580 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:48.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:48.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:48.582 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:56:48.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:56:48.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:48.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:49.235 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:50.237 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:50.237 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:50.237 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:56:50.238 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:56:50.238 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:50.238 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:51.145 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:56:52.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:56:52.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:56:52.147 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:56:52.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:56:52.148 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:56:52.148 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:56:52.584 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1389739416,v1:127.0.0.1:6819/1389739416] [v2:127.0.0.1:6820/1389739416,v1:127.0.0.1:6821/1389739416] exists,up 9714de9c-69be-4472-ac9c-f5abdc9becb2 2024-02-11T22:56:52.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:56:52.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:56:52.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:56:52.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-02-11T22:56:53.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-02-11T22:56:53.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-02-11T22:56:53.831 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'foo' already exists 2024-02-11T22:56:53.849 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-11T22:56:54.851 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-02-11T22:56:56.058 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 min_size to 1 2024-02-11T22:56:56.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-02-11T22:56:57.342 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 min_size to 2 2024-02-11T22:56:57.364 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-02-11T22:57:02.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-02-11T22:57:02.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:57:02.367 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:02.368 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:02.368 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:02.808 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1285530072,v1:127.0.0.1:6801/1285530072] [v2:127.0.0.1:6802/1285530072,v1:127.0.0.1:6803/1285530072] exists,up 475fb097-e760-4472-93b4-714d3615e74d 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:57:02.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:02.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:03.253 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1248167582,v1:127.0.0.1:6811/1248167582] [v2:127.0.0.1:6812/1248167582,v1:127.0.0.1:6813/1248167582] exists,up 003f4496-7202-4368-8681-eef69f484826 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-02-11T22:57:03.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:57:03.255 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:03.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:03.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:57:03.690 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1389739416,v1:127.0.0.1:6819/1389739416] [v2:127.0.0.1:6820/1389739416,v1:127.0.0.1:6821/1389739416] exists,up 9714de9c-69be-4472-ac9c-f5abdc9becb2 2024-02-11T22:57:03.691 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:57:03.691 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:57:03.691 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:57:03.691 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:53.654+0000 7fc9531d8640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:54.302+0000 7fc9531d8640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:54.714+0000 7fc9531d8640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:55.750+0000 7fc9511d4640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:56.218+0000 7fc9511d4640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.694 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:56.558+0000 7fc9531d8640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.106+0000 7fc9511d4640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.226+0000 7fc9511d4640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.402+0000 7fc9531d8640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.498+0000 7fc9531d8640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.834+0000 7fc9531d8640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:58.326+0000 7fc9531d8640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-02-11T22:57:03.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-02-11T22:57:03.697 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:53.654+0000 7fc9531d8640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-02-11T22:57:03.697 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:54.302+0000 7fc9531d8640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-02-11T22:57:03.697 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:54.714+0000 7fc9531d8640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-02-11T22:57:03.697 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:55.750+0000 7fc9511d4640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:56.218+0000 7fc9511d4640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:56.558+0000 7fc9531d8640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.106+0000 7fc9511d4640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.226+0000 7fc9511d4640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.406+0000 7fc9531d8640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.498+0000 7fc9531d8640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:57.834+0000 7fc9531d8640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-02-11T22:57:03.698 INFO:tasks.workunit.client.0.smithi121.stdout:2024-02-11T22:56:58.330+0000 7fc9531d8640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-02-11T22:57:03.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-02-11T22:57:03.699 INFO:tasks.workunit.client.0.smithi121.stdout:success 2024-02-11T22:57:03.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-02-11T22:57:03.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-02-11T22:57:03.700 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-02-11T22:57:04.213 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'foo' does not exist 2024-02-11T22:57:04.228 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-02-11T22:57:04.229 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:57:04.229 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:57:04.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:57:04.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:57:04.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:57:05.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:57:05.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-02-11T22:57:05.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-02-11T22:57:05.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T22:57:05.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-02-11T22:57:05.544 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:57:05.544 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:57:05.544 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:57:05.544 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:57:05.544 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:57:05.551 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:57:05.551 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T22:57:05.552 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T22:57:05.552 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T22:57:05.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T22:57:05.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T22:57:05.556 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T22:57:05.556 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.556 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T22:57:05.556 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T22:57:05.557 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.558 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T22:57:05.558 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-11T22:57:05.558 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-02-11T22:57:05.571 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T22:57:05.571 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:05.571 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:57:05.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20211 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/bad-inc-map 0 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:57:05.573 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:57:05.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:57:05.575 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T22:57:05.575 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T22:57:05.576 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T22:57:05.577 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T22:57:05.577 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T22:57:05.577 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T22:57:05.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T22:57:05.578 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T22:57:05.578 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T22:57:05.580 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-11T22:57:05.580 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-02-11T22:57:05.581 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T22:57:05.581 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:05.581 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20211 2024-02-11T22:57:05.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20211 2024-02-11T22:57:05.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T22:57:05.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T22:57:05.583 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-11T22:57:05.583 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-11T22:57:05.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-11T22:57:05.641 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-02-11T22:57:05.641 DEBUG:teuthology.orchestra.run.smithi121:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-02-11T22:57:05.694 INFO:tasks.workunit.client.0.smithi121.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-11T22:57:05.700 INFO:tasks.workunit.client.0.smithi121.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-11T22:57:05.700 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/divergent-priors 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-02-11T22:57:05.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-02-11T22:57:05.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-02-11T22:57:05.702 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-02-11T22:57:05.702 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-02-11T22:57:05.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-02-11T22:57:05.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-02-11T22:57:05.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-02-11T22:57:05.703 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-02-11T22:57:05.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-02-11T22:57:05.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-02-11T22:57:05.705 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-02-11T22:57:05.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none ' 2024-02-11T22:57:05.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-02-11T22:57:05.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-02-11T22:57:05.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-02-11T22:57:05.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:57:05.708 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-02-11T22:57:05.708 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-11T22:57:05.711 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:TEST_divergent_2 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:TEST_divergent_3 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:TEST_divergent_ec' 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:57:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:57:05.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:57:05.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:57:05.714 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T22:57:05.714 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T22:57:05.715 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T22:57:05.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T22:57:05.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T22:57:05.716 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T22:57:05.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T22:57:05.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T22:57:05.718 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T22:57:05.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T22:57:05.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-11T22:57:05.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-02-11T22:57:05.721 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T22:57:05.721 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:05.721 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:05.721 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26067 2024-02-11T22:57:05.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T22:57:05.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T22:57:05.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-02-11T22:57:05.723 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-11T22:57:05.724 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:05.724 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:05.724 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26067 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-02-11T22:57:05.726 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-02-11T22:57:05.727 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-02-11T22:57:05.727 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-02-11T22:57:05.727 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-02-11T22:57:05.727 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-11T22:57:05.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-02-11T22:57:05.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-02-11T22:57:06.149 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-11T22:57:06.149 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:57:06.149 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:57:06.149 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:57:06.149 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:06.150 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:06.150 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:06.150 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-11T22:57:06.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-11T22:57:06.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-11T22:57:06.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T22:57:06.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T22:57:06.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:06.371 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-mon.a.asok 2024-02-11T22:57:06.372 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T22:57:06.372 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26067/ceph-mon.a.asok config get fsid 2024-02-11T22:57:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-11T22:57:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T22:57:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T22:57:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:57:06.488 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:06.489 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:06.489 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-mon.a.asok 2024-02-11T22:57:06.489 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T22:57:06.489 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26067/ceph-mon.a.asok config get mon_host 2024-02-11T22:57:06.596 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-02-11T22:57:06.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:07.501 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:07.502 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T22:57:07.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T22:57:07.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-02-11T22:57:07.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-02-11T22:57:07.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T22:57:07.569 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:57:07.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T22:57:07.571 INFO:tasks.workunit.client.0.smithi121.stdout:add osd0 87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T22:57:07.572 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 87f810a3-99d5-4366-ba65-5afc3ca69433' 2024-02-11T22:57:07.572 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:57:07.593 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDUMll9n5DIxAASC/8Ss29IZzukhwd+iiU7A== 2024-02-11T22:57:07.593 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDUMll9n5DIxAASC/8Ss29IZzukhwd+iiU7A=="}' 2024-02-11T22:57:07.593 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87f810a3-99d5-4366-ba65-5afc3ca69433 -i td/divergent-priors/0/new.json 2024-02-11T22:57:07.892 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:07.908 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-02-11T22:57:07.910 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDUMll9n5DIxAASC/8Ss29IZzukhwd+iiU7A== --osd-uuid 87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T22:57:07.954 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.950+0000 7f886bf12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:07.958 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.954+0000 7f886bf12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:07.959 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.958+0000 7f886bf12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:07.960 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.958+0000 7f886bf12640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T22:57:07.960 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.958+0000 7f886bf12640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T22:57:07.960 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.958+0000 7f886bf12640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T22:57:07.998 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:07.994+0000 7f886bf12640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-02-11T22:57:14.152 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-02-11T22:57:14.152 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:57:14.153 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd0 key to auth repository 2024-02-11T22:57:14.154 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-11T22:57:14.154 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:57:14.709 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.0 2024-02-11T22:57:14.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-11T22:57:14.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:57:14.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:57:14.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:57:14.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:57:14.752 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:14.750+0000 7fccffecc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:14.775 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:14.774+0000 7fccffecc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:14.776 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:14.774+0000 7fccffecc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:15.150 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-11T22:57:15.150 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:57:15.150 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:57:15.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:15.152 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:15.332 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:15.330+0000 7fccffecc640 -1 Falling back to public interface 2024-02-11T22:57:15.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:16.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:16.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:16.584 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:57:16.585 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:57:16.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:16.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:16.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:17.941 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:17.938+0000 7fccffecc640 -1 osd.0 0 log_to_monitors true 2024-02-11T22:57:17.987 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:17.987 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:17.987 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:57:17.987 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:57:17.987 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:17.988 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:18.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:19.428 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:19.428 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:19.428 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:57:19.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:57:19.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:19.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:19.940 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:20.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:20.942 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:20.942 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:57:20.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:57:20.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:20.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:21.364 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:22.365 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:22.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:22.366 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:57:22.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:57:22.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:22.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:22.932 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:23.933 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:23.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:23.934 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:57:23.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:57:23.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:23.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:24.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:25.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:25.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:25.410 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:57:25.410 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:57:25.410 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:25.410 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:25.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:26.982 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:26.983 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:26.983 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:57:26.983 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:57:26.983 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:26.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:27.413 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:28.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:28.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:28.414 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:57:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:57:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:28.922 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:29.936 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:29.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:29.937 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:57:29.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:57:29.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:29.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:30.357 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:31.359 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:31.360 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:31.360 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:57:31.360 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:57:31.360 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:31.360 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:32.515 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:33.517 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:33.517 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:33.517 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:57:33.518 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:57:33.518 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:33.518 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:34.427 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:35.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:35.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:35.429 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:57:35.429 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:57:35.430 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:35.430 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:36.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:37.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:37.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:37.197 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:57:37.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:57:37.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:37.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:38.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:39.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:39.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:39.006 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:57:39.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:57:39.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:39.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:39.761 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:40.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:40.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:40.763 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:57:40.764 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:57:40.764 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:40.764 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:41.673 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:42.675 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:42.675 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:42.675 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:57:42.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:57:42.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:42.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:57:43.154 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1318773674,v1:127.0.0.1:6803/1318773674] [v2:127.0.0.1:6804/1318773674,v1:127.0.0.1:6805/1318773674] exists,up 87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T22:57:43.160 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:57:43.160 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:57:43.160 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T22:57:43.161 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:57:43.162 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-02-11T22:57:43.163 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:57:43.164 INFO:tasks.workunit.client.0.smithi121.stdout:add osd1 20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T22:57:43.164 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T22:57:43.164 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 20416f3e-9b4f-4ac5-8477-c61a1bc7990c' 2024-02-11T22:57:43.164 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:57:43.187 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnUMllx1AOCxAAOk98snhSv/rFPAt4LMOkyw== 2024-02-11T22:57:43.187 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnUMllx1AOCxAAOk98snhSv/rFPAt4LMOkyw=="}' 2024-02-11T22:57:43.187 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20416f3e-9b4f-4ac5-8477-c61a1bc7990c -i td/divergent-priors/1/new.json 2024-02-11T22:57:43.690 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:57:43.706 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-02-11T22:57:43.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnUMllx1AOCxAAOk98snhSv/rFPAt4LMOkyw== --osd-uuid 20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T22:57:43.741 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.738+0000 7f606629f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:43.754 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.750+0000 7f606629f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:43.755 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.754+0000 7f606629f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:43.755 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.754+0000 7f606629f640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T22:57:43.755 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.754+0000 7f606629f640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T22:57:43.756 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.754+0000 7f606629f640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T22:57:43.794 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:43.790+0000 7f606629f640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-02-11T22:57:50.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-02-11T22:57:50.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:57:50.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-11T22:57:50.125 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd1 key to auth repository 2024-02-11T22:57:50.125 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:57:50.675 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T22:57:50.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-11T22:57:50.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:57:50.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:57:50.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:57:50.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:57:50.714 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:50.714+0000 7f59a5ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:50.734 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:50.734+0000 7f59a5ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:50.736 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:50.734+0000 7f59a5ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:51.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:57:51.093 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:57:51.093 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:51.093 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:51.304 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:51.302+0000 7f59a5ac6640 -1 Falling back to public interface 2024-02-11T22:57:51.546 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:52.547 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:52.548 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:57:52.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:52.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:57:52.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:52.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:53.028 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:53.899 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:57:53.897+0000 7f59a5ac6640 -1 osd.1 0 log_to_monitors true 2024-02-11T22:57:54.029 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:54.029 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:54.030 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:57:54.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:57:54.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:54.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:54.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:55.604 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:55.604 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:55.604 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:57:55.605 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:57:55.605 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:55.605 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:56.016 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:57.018 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:57.018 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:57.018 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:57:57.019 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:57:57.019 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:57.019 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:57.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:58.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:58.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:58.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:57:58.447 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:57:58.447 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:58.447 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:57:58.979 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:57:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:57:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:57:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:57:59.981 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:57:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:57:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:00.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:01.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:01.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:01.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:58:01.385 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:58:01.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:01.386 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:01.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:02.940 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:02.940 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:02.940 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:58:02.940 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:58:02.941 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:02.941 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:03.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:04.358 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:04.358 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:04.358 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:58:04.359 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:58:04.359 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:04.359 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:04.918 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:05.920 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:05.920 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:05.920 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:58:05.920 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:58:05.921 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:05.921 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:06.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:07.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:07.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:07.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:58:07.351 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:58:07.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:07.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:08.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:09.091 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:09.091 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:09.091 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:58:09.092 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:58:09.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:09.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:09.812 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:10.814 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:10.814 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:10.814 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:58:10.814 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:58:10.815 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:10.815 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:12.066 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:13.067 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:13.068 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:13.068 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:58:13.068 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:58:13.068 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:13.069 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:13.886 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:14.888 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:14.888 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:14.888 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:58:14.888 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:58:14.889 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:14.889 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:15.990 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:16.991 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:16.991 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:16.992 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:58:16.992 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:58:16.992 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:16.992 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:17.922 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:18.923 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:18.923 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:18.924 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:58:18.924 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:58:18.924 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:18.924 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T22:58:19.384 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3380813962,v1:127.0.0.1:6811/3380813962] [v2:127.0.0.1:6812/3380813962,v1:127.0.0.1:6813/3380813962] exists,up 20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:58:19.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:58:19.391 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:58:19.392 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T22:58:19.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1455cafd-941b-4258-b66b-61b15fc14daf' 2024-02-11T22:58:19.394 INFO:tasks.workunit.client.0.smithi121.stdout:add osd2 1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T22:58:19.394 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T22:58:19.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALUclleFelGBAAMftEgptI3P1iwHhSSnfhPQ== 2024-02-11T22:58:19.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALUclleFelGBAAMftEgptI3P1iwHhSSnfhPQ=="}' 2024-02-11T22:58:19.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1455cafd-941b-4258-b66b-61b15fc14daf -i td/divergent-priors/2/new.json 2024-02-11T22:58:19.890 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:58:19.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-02-11T22:58:19.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALUclleFelGBAAMftEgptI3P1iwHhSSnfhPQ== --osd-uuid 1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T22:58:19.955 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.953+0000 7f4e3b053640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:19.957 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.953+0000 7f4e3b053640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.953+0000 7f4e3b053640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.957+0000 7f4e3b053640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T22:58:19.958 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.957+0000 7f4e3b053640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T22:58:19.959 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.957+0000 7f4e3b053640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T22:58:20.000 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:19.997+0000 7f4e3b053640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-02-11T22:58:26.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-02-11T22:58:26.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T22:58:26.591 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd2 key to auth repository 2024-02-11T22:58:26.592 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-11T22:58:26.592 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T22:58:27.142 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.2 2024-02-11T22:58:27.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-11T22:58:27.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:58:27.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T22:58:27.144 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T22:58:27.144 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T22:58:27.182 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:27.181+0000 7fc219ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:27.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:27.197+0000 7fc219ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:27.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:27.197+0000 7fc219ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:58:27.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-11T22:58:27.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:58:27.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T22:58:27.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:27.638 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:27.780 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:27.777+0000 7fc219ac8640 -1 Falling back to public interface 2024-02-11T22:58:28.029 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:29.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:29.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:29.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:58:29.031 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:58:29.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:29.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:29.433 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:30.435 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:30.436 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:30.436 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:58:30.436 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:58:30.436 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:30.436 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:30.458 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:58:30.457+0000 7fc219ac8640 -1 osd.2 0 log_to_monitors true 2024-02-11T22:58:31.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:32.141 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:32.142 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:58:32.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:32.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:58:32.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:32.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:32.649 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:33.651 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:33.651 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:33.651 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T22:58:33.652 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T22:58:33.652 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:33.652 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:34.071 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:35.073 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:35.073 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:35.074 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T22:58:35.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T22:58:35.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:35.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:35.599 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:36.601 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:36.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:36.602 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T22:58:36.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T22:58:36.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:36.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:37.047 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:38.049 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:38.049 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:38.049 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T22:58:38.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T22:58:38.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:38.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:38.645 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:39.646 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:39.647 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T22:58:39.647 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:39.647 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T22:58:39.647 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:39.648 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:40.072 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:41.073 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:41.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:41.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T22:58:41.074 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T22:58:41.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:41.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:41.553 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:42.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:42.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:42.555 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T22:58:42.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T22:58:42.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:42.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:43.163 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:44.165 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:44.165 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:44.165 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T22:58:44.165 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T22:58:44.166 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:44.166 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:45.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:46.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:46.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:46.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T22:58:46.293 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T22:58:46.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:46.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:46.965 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:47.967 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:47.967 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:47.967 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T22:58:47.967 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T22:58:47.967 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:47.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:48.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:49.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:49.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:49.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T22:58:49.713 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T22:58:49.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:49.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:50.265 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:51.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:51.267 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:51.267 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T22:58:51.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T22:58:51.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:51.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:51.915 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:52.917 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:52.917 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:52.917 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T22:58:52.918 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T22:58:52.918 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:52.918 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:53.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:54.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:54.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:54.570 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T22:58:54.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T22:58:54.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:54.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:55.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:58:56.024 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:58:56.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:58:56.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-11T22:58:56.025 INFO:tasks.workunit.client.0.smithi121.stdout:18 2024-02-11T22:58:56.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:58:56.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:58:56.425 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1138414549,v1:127.0.0.1:6819/1138414549] [v2:127.0.0.1:6820/1138414549,v1:127.0.0.1:6821/1138414549] exists,up 1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T22:58:56.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:58:56.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:58:56.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:58:56.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-02-11T22:58:57.504 INFO:tasks.workunit.client.0.smithi121.stderr:noout is set 2024-02-11T22:58:57.523 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-02-11T22:58:58.248 INFO:tasks.workunit.client.0.smithi121.stderr:noin is set 2024-02-11T22:58:58.267 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-02-11T22:58:58.910 INFO:tasks.workunit.client.0.smithi121.stderr:nodown is set 2024-02-11T22:58:58.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-02-11T22:58:58.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-11T22:58:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'test' already exists 2024-02-11T22:58:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-11T22:59:00.833 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-02-11T22:59:01.715 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 size to 3 2024-02-11T22:59:01.741 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-02-11T22:59:02.494 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 min_size to 2 2024-02-11T22:59:02.512 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-02-11T22:59:02.512 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T22:59:02.513 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T22:59:02.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T22:59:02.936 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T22:59:02.936 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T22:59:02.936 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T22:59:02.936 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:02.936 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T22:59:03.102 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-11T22:59:03.102 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-11T22:59:03.102 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-11T22:59:03.102 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:03.102 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T22:59:03.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-11T22:59:03.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-11T22:59:03.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672974' 2024-02-11T22:59:03.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:03.266 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T22:59:03.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-02-11T22:59:03.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-02-11T22:59:03.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672974 2-64424509444' 2024-02-11T22:59:03.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:03.415 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-11T22:59:03.415 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:03.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T22:59:03.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-11T22:59:03.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:03.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-11T22:59:03.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-11T22:59:03.417 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836505 2024-02-11T22:59:03.418 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:03.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836505 2024-02-11T22:59:03.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T22:59:04.824 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T22:59:04.825 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:05.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-11T22:59:05.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:05.227 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-11T22:59:05.227 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:05.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T22:59:05.230 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-11T22:59:05.230 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:05.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-11T22:59:05.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-11T22:59:05.231 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672974 2024-02-11T22:59:05.232 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T22:59:05.618 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-02-11T22:59:05.618 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:05.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-02-11T22:59:05.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:05.620 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T22:59:05.621 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:05.621 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-02-11T22:59:05.622 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-02-11T22:59:05.622 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-02-11T22:59:05.622 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509444 2024-02-11T22:59:05.623 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T22:59:06.031 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-11T22:59:06.032 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T22:59:06.032 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T22:59:06.033 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T22:59:06.033 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T22:59:06.033 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T22:59:06.133 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:06.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T22:59:06.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-02-11T22:59:06.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-02-11T22:59:06.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-02-11T22:59:06.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:06.697 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T22:59:06.852 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-11T22:59:06.852 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-11T22:59:06.853 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672977' 2024-02-11T22:59:06.853 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:06.853 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T22:59:07.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-11T22:59:07.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-11T22:59:07.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672977 2-64424509447' 2024-02-11T22:59:07.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:07.005 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:07.005 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-02-11T22:59:07.005 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T22:59:07.006 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-02-11T22:59:07.006 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:07.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-02-11T22:59:07.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-02-11T22:59:07.008 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836507 2024-02-11T22:59:07.008 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:07.397 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836507 2024-02-11T22:59:07.397 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T22:59:08.398 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T22:59:08.399 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:08.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836507 2024-02-11T22:59:08.818 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:08.819 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-11T22:59:08.819 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:08.819 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T22:59:08.821 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-11T22:59:08.821 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:08.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-11T22:59:08.822 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672977 2024-02-11T22:59:08.823 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-11T22:59:08.823 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T22:59:09.219 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-11T22:59:09.219 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:09.220 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-11T22:59:09.220 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:09.221 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T22:59:09.222 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-11T22:59:09.222 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:09.223 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-11T22:59:09.223 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-11T22:59:09.223 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509447 2024-02-11T22:59:09.224 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T22:59:09.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-11T22:59:09.626 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T22:59:09.626 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T22:59:09.626 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T22:59:10.169 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T22:59:10.169 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T22:59:10.170 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T22:59:10.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-11T22:59:10.589 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T22:59:10.589 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T22:59:10.590 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T22:59:11.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-11T22:59:11.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-11T22:59:11.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-11T22:59:11.158 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-02-11T22:59:11.158 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-02-11T22:59:11.585 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T22:59:11.602 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-02-11T22:59:11.603 INFO:tasks.workunit.client.0.smithi121.stdout:primary and soon to be divergent is 1 2024-02-11T22:59:11.603 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-02-11T22:59:11.603 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-02-11T22:59:12.022 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T22:59:12.022 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-11T22:59:00.994465+0000 0'0 29:14 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T08:00:37.510600+0000 0 0 2024-02-11T22:59:12.022 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T22:59:12.022 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T22:59:12.022 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T22:59:12.039 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-02-11T22:59:12.039 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-02-11T22:59:12.040 INFO:tasks.workunit.client.0.smithi121.stdout:writing initial objects 2024-02-11T22:59:12.041 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-02-11T22:59:12.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-02-11T22:59:12.244 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.244 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-02-11T22:59:12.412 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.412 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-02-11T22:59:12.587 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.588 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-02-11T22:59:12.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-02-11T22:59:12.932 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:12.932 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-02-11T22:59:13.099 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.099 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-02-11T22:59:13.267 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.267 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-02-11T22:59:13.434 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.435 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-02-11T22:59:13.610 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.610 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-02-11T22:59:13.761 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.761 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-02-11T22:59:13.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:13.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-02-11T22:59:14.104 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.104 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-02-11T22:59:14.280 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.280 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-02-11T22:59:14.449 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.449 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-02-11T22:59:14.623 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.623 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-02-11T22:59:14.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.799 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-02-11T22:59:14.959 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:14.959 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-02-11T22:59:15.109 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:15.109 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-02-11T22:59:15.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:15.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-02-11T22:59:15.461 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:15.461 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-02-11T22:59:15.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:15.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-02-11T22:59:15.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:15.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-02-11T22:59:16.014 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.014 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-02-11T22:59:16.207 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.207 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-02-11T22:59:16.374 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.374 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-02-11T22:59:16.534 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.534 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-02-11T22:59:16.709 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-02-11T22:59:16.895 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:16.895 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-02-11T22:59:17.069 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.069 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-02-11T22:59:17.237 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.237 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-02-11T22:59:17.413 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.413 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-02-11T22:59:17.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-02-11T22:59:17.773 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.773 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-02-11T22:59:17.932 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:17.933 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-02-11T22:59:18.111 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.111 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-02-11T22:59:18.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-02-11T22:59:18.460 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.460 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-02-11T22:59:18.636 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.637 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-02-11T22:59:18.812 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.812 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-02-11T22:59:18.979 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:18.980 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-02-11T22:59:19.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-02-11T22:59:19.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.315 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-02-11T22:59:19.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-02-11T22:59:19.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.650 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-02-11T22:59:19.825 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.825 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-02-11T22:59:19.976 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:19.976 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-02-11T22:59:20.127 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.127 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-02-11T22:59:20.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-02-11T22:59:20.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.471 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-02-11T22:59:20.647 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.647 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-02-11T22:59:20.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-02-11T22:59:20.973 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:20.973 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-02-11T22:59:21.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-02-11T22:59:21.292 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.292 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-02-11T22:59:21.442 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.443 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-02-11T22:59:21.610 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.610 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-02-11T22:59:21.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-02-11T22:59:21.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:21.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-02-11T22:59:22.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-02-11T22:59:22.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-02-11T22:59:22.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-02-11T22:59:22.668 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.668 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-02-11T22:59:22.833 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.834 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-02-11T22:59:22.986 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:22.986 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-02-11T22:59:23.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:23.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-02-11T22:59:23.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:23.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-02-11T22:59:23.487 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:23.487 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-02-11T22:59:23.672 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.253 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-02-11T22:59:24.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-02-11T22:59:24.254 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-02-11T22:59:24.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-02-11T22:59:24.350 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.350 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-02-11T22:59:24.501 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.501 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-02-11T22:59:24.668 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.668 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-02-11T22:59:24.878 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:24.878 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-02-11T22:59:25.045 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.045 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-02-11T22:59:25.221 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.222 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-02-11T22:59:25.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-02-11T22:59:25.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-02-11T22:59:25.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.707 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-02-11T22:59:25.858 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:25.858 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-02-11T22:59:26.009 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.009 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-02-11T22:59:26.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-02-11T22:59:26.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-02-11T22:59:26.521 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.521 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-02-11T22:59:26.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.705 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-02-11T22:59:26.923 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:26.923 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-02-11T22:59:27.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:27.090 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-02-11T22:59:27.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:27.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-02-11T22:59:27.408 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:27.408 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-02-11T22:59:27.592 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:27.592 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-02-11T22:59:28.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-02-11T22:59:28.211 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.211 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-02-11T22:59:28.386 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.386 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-02-11T22:59:28.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-02-11T22:59:28.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-02-11T22:59:28.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:28.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-02-11T22:59:29.108 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:29.108 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-02-11T22:59:29.259 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-02-11T22:59:29.259 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-02-11T22:59:29.418 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-02-11T22:59:29.418 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-02-11T22:59:29.418 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T22:59:29.419 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T22:59:29.419 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T22:59:29.419 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-02-11T22:59:29.420 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T22:59:29.420 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T22:59:29.420 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T22:59:29.420 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T22:59:29.420 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T22:59:29.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:29.912 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T22:59:30.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-02-11T22:59:30.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-02-11T22:59:30.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-02-11T22:59:30.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:30.081 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T22:59:30.241 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-11T22:59:30.241 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-11T22:59:30.241 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672984' 2024-02-11T22:59:30.241 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T22:59:30.241 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T22:59:30.400 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-11T22:59:30.401 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-11T22:59:30.401 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672984 2-64424509453' 2024-02-11T22:59:30.401 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:30.401 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-02-11T22:59:30.401 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:30.403 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T22:59:30.403 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-02-11T22:59:30.403 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:30.404 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-02-11T22:59:30.405 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-02-11T22:59:30.405 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836514 2024-02-11T22:59:30.405 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:30.855 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836514 2024-02-11T22:59:30.855 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T22:59:31.857 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T22:59:31.858 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T22:59:32.293 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-02-11T22:59:32.294 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:32.295 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-11T22:59:32.295 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:32.296 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T22:59:32.297 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-11T22:59:32.297 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:32.298 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672984 2024-02-11T22:59:32.299 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-11T22:59:32.299 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-11T22:59:32.299 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T22:59:32.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-11T22:59:32.756 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T22:59:32.757 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-11T22:59:32.757 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T22:59:32.758 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T22:59:32.759 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-11T22:59:32.759 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T22:59:32.760 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-11T22:59:32.760 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-11T22:59:32.761 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509453 2024-02-11T22:59:32.761 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T22:59:33.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-02-11T22:59:33.180 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T22:59:33.181 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T22:59:33.181 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T22:59:33.700 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T22:59:33.700 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T22:59:33.701 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T22:59:33.701 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T22:59:33.701 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T22:59:33.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T22:59:33.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T22:59:33.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T22:59:34.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-11T22:59:34.099 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T22:59:34.099 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T22:59:34.099 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T22:59:34.629 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-11T22:59:34.629 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-11T22:59:34.629 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-11T22:59:34.630 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-02-11T22:59:34.630 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-02-11T22:59:34.630 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-02-11T22:59:34.630 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-02-11T22:59:34.630 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-11T22:59:35.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-02-11T22:59:35.043 INFO:tasks.workunit.client.0.smithi121.stdout:blackholing osds 0 2 2024-02-11T22:59:35.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-02-11T22:59:35.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-02-11T22:59:35.460 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T22:59:35.460 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-02-11T22:59:00.994465+0000 29'100 29:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T08:00:37.510600+0000 0 0 2024-02-11T22:59:35.460 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T22:59:35.460 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T22:59:35.461 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T22:59:35.474 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:35.475 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-02-11T22:59:35.476 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-11T22:59:35.476 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-11T22:59:35.476 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:59:35.476 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:35.477 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:35.477 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.0.asok 2024-02-11T22:59:35.477 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-02-11T22:59:35.477 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.0.asok config set objectstore_blackhole 1 2024-02-11T22:59:35.578 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T22:59:35.578 INFO:tasks.workunit.client.0.smithi121.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-02-11T22:59:35.578 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T22:59:35.590 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:35.591 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-02-11T22:59:35.591 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-02-11T22:59:35.591 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-02-11T22:59:35.592 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:59:35.592 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:35.592 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:35.593 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.2.asok 2024-02-11T22:59:35.593 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-02-11T22:59:35.593 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.2.asok config set objectstore_blackhole 1 2024-02-11T22:59:35.700 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T22:59:35.701 INFO:tasks.workunit.client.0.smithi121.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-02-11T22:59:35.701 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T22:59:35.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-02-11T22:59:35.714 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-02-11T22:59:35.715 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-02-11T22:59:35.715 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-02-11T22:59:35.715 INFO:tasks.workunit.client.0.smithi121.stdout:writing divergent object 2024-02-11T22:59:35.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-02-11T22:59:35.716 INFO:tasks.workunit.client.0.smithi121.stdout:create missing divergent object 2024-02-11T22:59:35.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=test 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=existing_99 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-11T22:59:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/divergent-priors 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds test existing_99 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-02-11T22:59:35.719 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-11T22:59:35.720 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:0 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0' '2') 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-11T22:59:36.120 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-11T22:59:36.121 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/divergent-priors/1/type 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-11T22:59:36.122 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-11T22:59:36.123 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-11T22:59:36.123 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-11T22:59:36.123 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-11T22:59:36.123 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:59:36.123 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:36.124 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:36.124 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.1.asok 2024-02-11T22:59:36.124 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.26067/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-11T22:59:36.246 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-11T22:59:36.246 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-11T22:59:36.247 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-11T22:59:36.247 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-11T22:59:36.247 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-11T22:59:36.247 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.1.asok 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-11T22:59:36.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.1.asok injectdataerr test existing_99 2024-02-11T22:59:36.368 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-02-11T22:59:36.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-02-11T22:59:46.370 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-02-11T22:59:46.387 INFO:tasks.workunit.client.0.smithi121.stderr:rados: no process found 2024-02-11T22:59:46.387 INFO:tasks.workunit.client.0.smithi121.stdout:killing all the osds 2024-02-11T22:59:46.388 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-02-11T22:59:46.388 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-02-11T22:59:46.778 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T22:59:46.778 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-02-11T22:59:36.553492+0000 29'101 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T08:00:37.510600+0000 0 0 2024-02-11T22:59:46.778 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T22:59:46.779 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T22:59:46.779 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T22:59:46.794 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-02-11T22:59:46.794 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T22:59:46.794 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T22:59:46.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T22:59:46.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T22:59:46.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T22:59:46.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T22:59:46.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-02-11T22:59:46.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-02-11T22:59:47.429 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already down. 2024-02-11T22:59:47.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-02-11T22:59:47.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-02-11T22:59:48.054 INFO:tasks.workunit.client.0.smithi121.stderr:osd.1 is already down. 2024-02-11T22:59:48.069 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-02-11T22:59:48.070 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-02-11T22:59:48.734 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already down. 2024-02-11T22:59:48.751 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:48.751 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-02-11T22:59:49.311 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already out. 2024-02-11T22:59:49.327 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:49.327 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-02-11T22:59:49.862 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already out. 2024-02-11T22:59:49.878 INFO:tasks.workunit.client.0.smithi121.stdout:bringing up non_divergent 0 2 2024-02-11T22:59:49.879 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-02-11T22:59:49.879 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-02-11T22:59:50.286 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T22:59:50.286 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-02-11T22:59:36.553492+0000 29'101 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T08:00:37.510600+0000 0 0 2024-02-11T22:59:50.286 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T22:59:50.287 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T22:59:50.287 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T22:59:50.307 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:50.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-02-11T22:59:50.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T22:59:50.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T22:59:50.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-11T22:59:50.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:59:50.309 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:50.310 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:59:50.311 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-02-11T22:59:50.312 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.0 2024-02-11T22:59:50.313 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-11T22:59:50.313 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:59:50.314 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-02-11T22:59:50.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-11T22:59:50.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T22:59:50.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T22:59:50.314 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T22:59:50.348 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:50.345+0000 7f6b285ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:50.349 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:50.345+0000 7f6b285ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:50.351 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:50.349+0000 7f6b285ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:59:50.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:50.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:59:50.704 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:59:50.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:50.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:59:50.888 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:50.885+0000 7f6b285ef640 -1 Falling back to public interface 2024-02-11T22:59:51.121 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:52.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:59:52.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:52.123 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:59:52.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:59:52.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:52.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:59:52.552 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:53.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:59:53.554 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:53.554 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:59:53.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:59:53.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:53.555 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:59:53.936 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:53.933+0000 7f6b285ef640 -1 osd.0 29 log_to_monitors true 2024-02-11T22:59:54.058 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:55.059 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:59:55.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:55.060 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T22:59:55.061 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T22:59:55.061 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:55.061 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T22:59:55.467 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2389664721,v1:127.0.0.1:6803/2389664721] [v2:127.0.0.1:6804/2389664721,v1:127.0.0.1:6805/2389664721] exists,up 87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-11T22:59:55.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T22:59:55.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T22:59:55.472 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T22:59:55.472 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T22:59:55.472 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T22:59:55.472 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:59:55.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-02-11T22:59:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-11T22:59:55.474 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.2 2024-02-11T22:59:55.474 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T22:59:55.474 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-02-11T22:59:55.474 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-11T22:59:55.475 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T22:59:55.475 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T22:59:55.475 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T22:59:55.518 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:55.516+0000 7fe366565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:55.520 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:55.516+0000 7fe366565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:55.521 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:55.516+0000 7fe366565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:55.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T22:59:55.893 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T22:59:55.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:55.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:59:56.083 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:56.080+0000 7fe366565640 -1 Falling back to public interface 2024-02-11T22:59:56.326 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:57.327 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:59:57.328 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:57.328 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T22:59:57.328 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T22:59:57.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:57.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:59:57.743 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:58.744 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T22:59:58.745 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T22:59:58.745 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T22:59:58.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T22:59:58.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T22:59:58.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T22:59:59.173 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T22:59:59.461 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T22:59:59.456+0000 7fe366565640 -1 osd.2 29 log_to_monitors true 2024-02-11T23:00:00.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:00:00.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:00:00.175 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:00:00.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:00:00.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:00:00.177 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:00:00.580 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2353970367,v1:127.0.0.1:6811/2353970367] [v2:127.0.0.1:6812/2353970367,v1:127.0.0.1:6813/2353970367] exists,up 1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T23:00:00.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:00:00.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:00:00.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:00:00.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-02-11T23:00:00.581 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-02-11T23:00:01.109 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already in. 2024-02-11T23:00:01.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-02-11T23:00:01.124 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-02-11T23:00:01.646 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already in. 2024-02-11T23:00:01.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-02-11T23:00:01.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-02-11T23:00:01.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T23:00:01.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T23:00:01.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T23:00:01.663 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-02-11T23:00:01.664 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T23:00:01.664 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T23:00:01.664 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T23:00:01.664 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T23:00:01.664 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:00:01.725 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:00:02.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:00:02.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822660 2024-02-11T23:00:02.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822660 2024-02-11T23:00:02.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822660' 2024-02-11T23:00:02.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:00:02.362 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:00:02.476 INFO:tasks.workunit.client.0.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-11T23:00:02.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-02-11T23:00:02.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-02-11T23:00:02.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-02-11T23:00:02.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:00:02.478 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:00:02.639 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757251 2024-02-11T23:00:02.639 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757251 2024-02-11T23:00:02.639 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-02-11T23:00:02.639 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:00:02.640 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-154618822660 2024-02-11T23:00:02.640 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:00:02.640 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:00:02.641 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-154618822660 2024-02-11T23:00:02.641 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:00:02.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822660 2024-02-11T23:00:02.642 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 154618822660 2024-02-11T23:00:02.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-02-11T23:00:02.643 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:00:03.039 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822658 -lt 154618822660 2024-02-11T23:00:03.039 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T23:00:04.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T23:00:04.042 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:00:04.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822660 -lt 154618822660 2024-02-11T23:00:04.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:00:04.464 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:00:04.464 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-163208757251 2024-02-11T23:00:04.465 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:00:04.466 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-163208757251 2024-02-11T23:00:04.466 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:00:04.467 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757251 2024-02-11T23:00:04.467 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-02-11T23:00:04.467 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 163208757251 2024-02-11T23:00:04.467 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:00:04.921 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757251 -lt 163208757251 2024-02-11T23:00:04.921 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T23:00:04.922 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:04.922 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:05.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T23:00:05.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:05.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:05.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:05.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:05.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:05.471 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:05.471 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:05.862 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:05.862 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:05.863 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:05.863 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:06.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:06.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-11T23:00:06.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-11T23:00:06.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-11T23:00:06.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:06.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-11T23:00:06.484 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:06.484 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:06.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:06.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:06.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:06.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:06.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:06.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:06.873 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:06.874 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:06.874 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:07.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:07.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:07.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:07.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:07.384 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:07.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:07.385 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:07.385 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:07.385 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:07.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:07.945 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:07.946 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-02-11T23:00:07.946 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:07.946 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-11T23:00:08.146 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:08.147 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:08.147 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:08.147 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:08.147 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:08.148 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:08.148 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:08.148 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:08.543 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:08.544 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:08.544 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:08.544 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:09.098 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:09.686 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:09.686 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:09.687 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-02-11T23:00:09.687 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:09.687 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-11T23:00:10.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:10.088 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:10.088 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:10.088 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:10.088 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:10.088 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:10.089 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:10.089 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:10.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:10.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:10.493 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:10.493 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:11.032 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:11.032 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:11.033 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:11.567 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:11.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:11.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-02-11T23:00:11.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:11.568 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-11T23:00:12.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:12.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:12.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:12.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:12.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:12.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:12.370 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:12.371 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:12.792 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:12.792 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:12.793 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:12.793 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:13.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:13.320 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:13.320 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:13.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:13.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:13.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-02-11T23:00:13.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:13.882 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-02-11T23:00:15.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:15.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:15.484 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:15.484 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:15.484 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:15.484 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:15.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:15.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:15.875 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:15.875 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:15.875 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:15.876 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:16.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:16.393 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:16.394 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:16.395 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:16.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:16.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:16.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-02-11T23:00:16.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:16.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-02-11T23:00:20.136 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:20.137 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:20.137 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:20.137 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:20.137 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:20.138 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:20.138 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:20.138 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:20.538 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:20.539 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:20.539 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:20.539 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:21.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:21.090 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:21.090 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:21.090 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:21.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:21.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:21.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-02-11T23:00:21.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:21.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-02-11T23:00:28.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:28.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:28.064 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:28.064 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:28.064 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:28.064 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:28.065 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:28.065 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:28.465 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:28.466 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:28.466 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:28.466 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:28.968 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:28.969 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:28.969 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:29.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:29.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:29.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-02-11T23:00:29.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:00:29.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-02-11T23:00:36.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:00:36.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:00:36.795 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:00:36.796 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:00:36.796 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:00:36.796 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:00:36.796 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:00:36.796 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:00:37.224 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:00:37.225 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:00:37.225 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:00:37.225 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:00:37.770 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:00:37.771 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:00:37.771 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:00:37.771 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:00:38.307 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:00:38.307 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:00:38.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-02-11T23:00:38.308 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-02-11T23:00:38.849 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_fingerprint": "71bef125-3dea-46d0-a4bb-b46194aa2e49", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "commit": "d4ed20cfb238ef2c47d64ca4533eca00b8968b98", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "timestamp": "2024-02-11T23:00:38.848317+0000", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "tag": "", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "health": { 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "status": "HEALTH_WARN", 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "checks": { 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "OSDMAP_FLAGS": { 2024-02-11T23:00:38.850 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "nodown,noout,noin flag(s) set", 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 17 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [], 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "OSD_DOWN": { 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 osds down", 2024-02-11T23:00:38.851 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "osd.1 (root=default,host=smithi121) is down" 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "PG_DEGRADED": { 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:00:38.852 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "POOL_APP_NOT_ENABLED": { 2024-02-11T23:00:38.853 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 pool(s) do not have an application enabled", 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "application not enabled on pool 'test'" 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.854 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "mutes": [] 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_first_committed": 1, 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_last_committed": 1, 2024-02-11T23:00:38.855 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap": { 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T22:57:05.780633Z", 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T22:57:05.780633Z", 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release": 19, 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release_name": "squid", 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "election_strategy": 1, 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "disallowed_leaders: ": "", 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": false, 2024-02-11T23:00:38.856 INFO:tasks.workunit.client.0.smithi121.stdout: "tiebreaker_mon": "", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_ranks: ": "", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "features": { 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "persistent": [ 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "kraken", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "luminous", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "mimic", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap-prune", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "nautilus", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "octopus", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "pacific", 2024-02-11T23:00:38.857 INFO:tasks.workunit.client.0.smithi121.stdout: "elector-pinging", 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "quincy", 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "reef", 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "squid" 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "optional": [] 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "mons": [ 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "rank": 0, 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "a", 2024-02-11T23:00:38.858 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115", 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 0 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115/0", 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:7115/0", 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "priority": 0, 2024-02-11T23:00:38.859 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 0, 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_location": "{}" 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: "quorum": [ 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: 0 2024-02-11T23:00:38.860 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap": { 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 41, 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T22:57:06.660338+0000", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:00:01.918247+0000", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "last_up_change": "2024-02-11T22:59:59.780971+0000", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "last_in_change": "2024-02-11T23:00:01.477105+0000", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-11T23:00:38.863 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_num": 5800384, 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_set": [ 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "nodown", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "noin", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "noout", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "pglog_hardlimit", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snapdirs", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "recovery_deletes", 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "sortbitwise" 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_version": 9, 2024-02-11T23:00:38.864 INFO:tasks.workunit.client.0.smithi121.stdout: "full_ratio": 0.99000000953674316, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_snapshot": "", 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_max": 1, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "max_osd": 3, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "min_compat_client": "jewel", 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "require_osd_release": "squid", 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "allow_crimson": false, 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: "pools": [ 2024-02-11T23:00:38.865 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_name": "test", 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "create_time": "2024-02-11T22:58:59.267841+0000", 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": 1, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_names": "hashpspool", 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "size": 3, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "min_size": 2, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_rule": 0, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-11T23:00:38.866 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "object_hash": 2, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num": 1, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num": 1, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num_target": 1, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_target": 1, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_pending": 1, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pg_merge_meta": { 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "source_pgid": "0.0", 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "ready_epoch": 0, 2024-02-11T23:00:38.867 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_started": 0, 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": 0, 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "source_version": "0'0", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "target_version": "0'0" 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "last_change": "29", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend": "0", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "auid": 0, 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-11T23:00:38.868 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_seq": 0, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_epoch": 0, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_snaps": [], 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps": "[]", 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_bytes": 0, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_objects": 0, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "tiers": [], 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "tier_of": -1, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "read_tier": -1, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "write_tier": -1, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_mode": "none", 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_bytes": 0, 2024-02-11T23:00:38.869 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_objects": 0, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profile": "", 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_params": { 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "none" 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_period": 0, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_count": 0, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "use_gmt_hitset": true, 2024-02-11T23:00:38.870 INFO:tasks.workunit.client.0.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "grade_table": [], 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "stripe_width": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "expected_num_objects": 0, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "fast_read": false, 2024-02-11T23:00:38.871 INFO:tasks.workunit.client.0.smithi121.stdout: "options": {}, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "application_metadata": {}, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "read_balance": { 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "score_acting": 2.9900000095367432, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "score_stable": 2.9900000095367432, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_score": 0.67000001668930054, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_acting": 2, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_stable": 2, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity_weighted": 1, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity": 1, 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity_weighted": 1 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.872 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "osds": [ 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "87f810a3-99d5-4366-ba65-5afc3ca69433", 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 5, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 29, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 36, 2024-02-11T23:00:38.873 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 40, 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 30, 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6802", 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6803", 2024-02-11T23:00:38.874 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6804", 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.875 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6805", 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6808", 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.876 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6809", 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6806", 2024-02-11T23:00:38.877 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6807", 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2389664721 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.878 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6803/2389664721", 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6805/2389664721", 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2389664721", 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2389664721", 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:00:38.879 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "20416f3e-9b4f-4ac5-8477-c61a1bc7990c", 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 0, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 0, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 0, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 10, 2024-02-11T23:00:38.880 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 23, 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 31, 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.881 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.882 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.883 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:00:38.884 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3380813962 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.885 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/3380813962", 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/3380813962", 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3380813962", 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3380813962", 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "exists" 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "1455cafd-941b-4258-b66b-61b15fc14daf", 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:00:38.886 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 15, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 29, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 38, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 0, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 32, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.887 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.888 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:00:38.889 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2353970367 2024-02-11T23:00:38.890 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/2353970367", 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/2353970367", 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2353970367", 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2353970367", 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_xinfo": [ 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.891 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T22:59:47.257207+0000", 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T22:57:41.668348+0000", 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T22:59:47.811142+0000", 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:00:38.892 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T22:58:17.624182+0000", 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T22:59:48.426742+0000", 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T22:58:54.635333+0000", 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.893 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap": [], 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_items": [], 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_primaries": [], 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_temp": [], 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_temp": [], 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "blocklist": {}, 2024-02-11T23:00:38.894 INFO:tasks.workunit.client.0.smithi121.stdout: "range_blocklist": {}, 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profiles": { 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "default": { 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "k": "2", 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "m": "2", 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "plugin": "jerasure", 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "technique": "reed_sol_van" 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.895 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps_queue": [], 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "new_removed_snaps": [], 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "new_purged_snaps": [], 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_node_flags": {}, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "device_class_flags": {}, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": { 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_metadata": [ 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.896 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6804/2389664721,v1:127.0.0.1:6805/2389664721]", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:00:38.897 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T22:57:13.471202Z", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:00:38.898 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6802/2389664721,v1:127.0.0.1:6803/2389664721]", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2389664721,v1:127.0.0.1:6809/2389664721]", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2389664721,v1:127.0.0.1:6807/2389664721]", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:00:38.899 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/0", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/3380813962,v1:127.0.0.1:6813/3380813962]", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:00:38.900 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.901 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T22:57:49.414632Z", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/3380813962,v1:127.0.0.1:6811/3380813962]", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3380813962,v1:127.0.0.1:6817/3380813962]", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3380813962,v1:127.0.0.1:6815/3380813962]", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:00:38.902 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/1", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/2353970367,v1:127.0.0.1:6813/2353970367]", 2024-02-11T23:00:38.903 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-02-11T23:00:38.904 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T22:58:25.896484Z", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:00:38.905 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/2353970367,v1:127.0.0.1:6811/2353970367]", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2353970367,v1:127.0.0.1:6817/2353970367]", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2353970367,v1:127.0.0.1:6815/2353970367]", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:00:38.906 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/2", 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_clean_epochs": { 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "min_last_epoch_clean": 0, 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": { 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "per_pool": [] 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_epochs": [ 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.907 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 36 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 10 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 38 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_first_committed": 1, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_last_committed": 41, 2024-02-11T23:00:38.908 INFO:tasks.workunit.client.0.smithi121.stdout: "crushmap": { 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": [ 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.0", 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.1", 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.2", 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:00:38.909 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "types": [ 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 0, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd" 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "host" 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 2, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "chassis" 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.910 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 3, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "rack" 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 4, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "row" 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 5, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pdu" 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 6, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pod" 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.911 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 7, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "room" 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 8, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "datacenter" 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 9, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "zone" 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 10, 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "region" 2024-02-11T23:00:38.912 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "root" 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "buckets": [ 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -1, 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default", 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:00:38.913 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -2, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default~hdd", 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.914 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121", 2024-02-11T23:00:38.915 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.916 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121~hdd", 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:00:38.917 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:00:38.918 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "rules": [ 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_id": 0, 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_name": "replicated_rule", 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "steps": [ 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "take", 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "item": -1, 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: "item_name": "default" 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.919 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "choose_firstn", 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 0, 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "osd" 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "emit" 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "tunables": { 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_tries": 0, 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_fallback_tries": 0, 2024-02-11T23:00:38.920 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_total_tries": 50, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_descend_once": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_vary_r": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_stable": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "straw_calc_version": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "allowed_bucket_algs": 54, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "profile": "jewel", 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_tunables": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "legacy_tunables": 0, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "minimum_required_version": "jewel", 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables2": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v2_rules": 0, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables3": 1, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v3_rules": 0, 2024-02-11T23:00:38.921 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v4_buckets": 1, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables5": 1, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v5_rules": 0 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_args": {} 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "fsmap": { 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "default_fscid": -1, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": { 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": {}, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "ro_compat": {}, 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "incompat": { 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_1": "base v0.20", 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_2": "client writeable ranges", 2024-02-11T23:00:38.922 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_3": "default file layouts on dirs", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_4": "dir inode in separate object", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_5": "mds uses versioned encoding", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_6": "dirfrag is stored in omap", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_8": "no anchor table", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_9": "file layout v2", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_10": "snaprealm v2", 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_11": "minor log segments" 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_flags": { 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "enable_multiple": true, 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "ever_enabled_multiple": true 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "standbys": [], 2024-02-11T23:00:38.923 INFO:tasks.workunit.client.0.smithi121.stdout: "filesystems": [] 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_first_committed": 1, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_last_committed": 1, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "auth": { 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 5, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_secrets": 3 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_active": 1, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_unknown": 0, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osd": 3, 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_sum": { 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:00:38.924 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 61700, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 100, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 100, 2024-02-11T23:00:38.925 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:00:38.926 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 0, 2024-02-11T23:00:38.927 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 16, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 16, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 0 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum": { 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:00:38.928 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83636, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2112, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489164, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036903936, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:00:38.929 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2162688, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 459940, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10395, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416933 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 1, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.930 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum_by_class": { 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "hdd": { 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:00:38.931 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83636, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2112, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489164, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036903936, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2162688, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 459940, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:00:38.932 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10395, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416933 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 1, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:00:38.933 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_stats": [ 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "poolid": 1, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 61700, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:00:38.934 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 100, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 100, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:00:38.935 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:00:38.936 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 1228800, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 185100, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:00:38.937 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 16, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 16, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 3 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_stats": [ 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 154618822667 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.938 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 42949672987 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 163208757258 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_state": [ 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "state": "active+undersized+degraded", 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 1 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.939 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_osd": [ 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 1, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:00:38.940 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 0, 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": [ 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": {} 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "servicemap": { 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 5, 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T22:59:50.036727+0000", 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "services": {} 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:00:38.941 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_first_committed": 1, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_last_committed": 95, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_first_committed": 1, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_last_committed": 391, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "paxos": { 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 567, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pn": 0, 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: "accepted_pn": 0 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:00:38.942 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stderr:report 1459938384 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-02-11T23:00:38.943 INFO:tasks.workunit.client.0.smithi121.stdout:writing non-divergent object existing_4 2024-02-11T23:00:38.944 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:39.271 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:39.271 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 29'100 41:121 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:39.272 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:39.272 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:39.272 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:39.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-02-11T23:00:39.403 INFO:tasks.workunit.client.0.smithi121.stdout:delay recovery 2024-02-11T23:00:39.403 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-02-11T23:00:39.403 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:39.793 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:39.793 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 29'100 41:121 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:39.793 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:39.793 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:39.793 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:39.807 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-02-11T23:00:39.808 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-02-11T23:00:39.808 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-11T23:00:39.808 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-11T23:00:39.809 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:00:39.809 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:00:39.809 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:00:39.809 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.0.asok 2024-02-11T23:00:39.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-02-11T23:00:39.810 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.0.asok set_recovery_delay 100000 2024-02-11T23:00:39.920 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:00:39.932 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-02-11T23:00:39.933 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-02-11T23:00:39.933 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-02-11T23:00:39.933 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-02-11T23:00:39.934 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:00:39.934 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:00:39.934 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:00:39.934 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.2.asok 2024-02-11T23:00:39.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-02-11T23:00:39.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.2.asok set_recovery_delay 100000 2024-02-11T23:00:40.041 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:00:40.054 INFO:tasks.workunit.client.0.smithi121.stdout:revive divergent 1 2024-02-11T23:00:40.055 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-02-11T23:00:40.055 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:40.453 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:40.453 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 29'100 41:121 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:40.453 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:40.453 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:40.453 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:40.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-02-11T23:00:41.203 INFO:tasks.workunit.client.0.smithi121.stderr:noup is set 2024-02-11T23:00:41.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-02-11T23:00:41.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:00:41.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:00:41.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-11T23:00:41.229 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:00:41.230 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:00:41.231 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:00:41.231 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:00:41.232 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-02-11T23:00:41.233 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T23:00:41.234 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-11T23:00:41.234 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:00:41.234 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-02-11T23:00:41.234 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-11T23:00:41.235 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:00:41.235 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:00:41.235 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:00:41.273 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:41.268+0000 7f7d94aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:41.275 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:41.272+0000 7f7d94aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:41.276 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:41.272+0000 7f7d94aff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:41.634 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-02-11T23:00:41.634 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-02-11T23:00:41.812 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:41.808+0000 7f7d94aff640 -1 Falling back to public interface 2024-02-11T23:00:44.972 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:44.968+0000 7f7d94aff640 -1 osd.1 29 log_to_monitors true 2024-02-11T23:00:46.636 INFO:tasks.workunit.client.0.smithi121.stdout:delay recovery divergent 2024-02-11T23:00:46.636 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-02-11T23:00:46.636 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:47.037 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:47.038 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 41'101 43:125 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:47.038 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:47.038 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:47.054 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-02-11T23:00:47.055 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-11T23:00:47.055 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.1.asok 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-02-11T23:00:47.056 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.1.asok set_recovery_delay 100000 2024-02-11T23:00:47.168 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:00:47.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-02-11T23:00:47.774 INFO:tasks.workunit.client.0.smithi121.stderr:noup is unset 2024-02-11T23:00:47.794 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-02-11T23:00:47.794 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:00:47.795 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:00:47.796 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:00:47.796 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:00:48.276 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2389664721,v1:127.0.0.1:6803/2389664721] [v2:127.0.0.1:6804/2389664721,v1:127.0.0.1:6805/2389664721] exists,up 87f810a3-99d5-4366-ba65-5afc3ca69433 2024-02-11T23:00:48.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:00:48.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:00:48.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:00:48.278 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:00:48.279 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:00:48.279 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:00:48.815 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1960011895,v1:127.0.0.1:6819/1960011895] [v2:127.0.0.1:6820/1960011895,v1:127.0.0.1:6821/1960011895] exists,up 20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:00:48.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:00:48.818 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:00:48.818 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:00:48.818 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:00:49.290 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2353970367,v1:127.0.0.1:6811/2353970367] [v2:127.0.0.1:6812/2353970367,v1:127.0.0.1:6813/2353970367] exists,up 1455cafd-941b-4258-b66b-61b15fc14daf 2024-02-11T23:00:49.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:00:49.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:00:49.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:00:49.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:49.682 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:49.683 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 41'101 43:125 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:49.683 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:49.683 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:49.683 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:49.698 INFO:tasks.workunit.client.0.smithi121.stdout:wait for peering 2024-02-11T23:00:49.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-02-11T23:00:49.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:50.093 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:50.093 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 41'101 43:125 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:50.093 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:50.093 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:50.093 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:50.110 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-02-11T23:00:50.262 INFO:tasks.workunit.client.0.smithi121.stdout:killing divergent 1 2024-02-11T23:00:50.263 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-02-11T23:00:50.263 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:50.647 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:50.647 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 41'101 43:125 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:50.647 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:50.647 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:50.647 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:50.662 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-02-11T23:00:50.662 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:00:50.662 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:00:50.663 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:00:50.663 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:00:50.663 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:00:50.771 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:00:50.771 INFO:tasks.workunit.client.0.smithi121.stdout:reviving divergent 1 2024-02-11T23:00:50.772 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-02-11T23:00:50.772 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-02-11T23:00:51.141 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:00:51.196 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:00:02.296228+0000 41'101 43:125 [0,2] 0 [0,2] 0 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:05:12.017096+0000 0 0 2024-02-11T23:00:51.196 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:00:51.196 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:00:51.198 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:00:51.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-02-11T23:00:51.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:00:51.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:00:51.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:00:51.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:00:51.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-02-11T23:00:51.203 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T23:00:51.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-11T23:00:51.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:51.196+0000 7f46d286a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:51.196+0000 7f46d286a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:51.205 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:51.200+0000 7f46d286a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:00:51.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:00:51.586 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:00:51.586 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:00:51.586 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:00:51.586 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:00:51.767 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:51.764+0000 7f46d286a640 -1 Falling back to public interface 2024-02-11T23:00:52.030 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1960011895,v1:127.0.0.1:6819/1960011895] [v2:127.0.0.1:6820/1960011895,v1:127.0.0.1:6821/1960011895] exists,up 20416f3e-9b4f-4ac5-8477-c61a1bc7990c 2024-02-11T23:00:52.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:00:52.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:00:52.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:00:52.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-02-11T23:00:54.879 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:00:54.876+0000 7f46d286a640 -1 osd.1 48 log_to_monitors true 2024-02-11T23:01:12.032 INFO:tasks.workunit.client.0.smithi121.stdout:allowing recovery 2024-02-11T23:01:12.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-02-11T23:01:12.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-02-11T23:01:12.436 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:01:12.436 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-02-11T23:00:56.838213+0000 48'102 53:151 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T09:33:29.063786+0000 0 0 2024-02-11T23:01:12.436 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:01:12.436 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:01:12.436 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:01:12.452 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-02-11T23:01:12.452 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-02-11T23:01:12.606 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:01:12.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-02-11T23:01:12.620 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-02-11T23:01:12.788 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:01:12.802 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-02-11T23:01:12.802 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-02-11T23:01:12.941 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:01:12.955 INFO:tasks.workunit.client.0.smithi121.stdout:reading divergent objects 2024-02-11T23:01:12.955 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-02-11T23:01:12.955 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-02-11T23:01:13.331 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:01:13.331 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-02-11T23:00:56.838213+0000 48'102 53:151 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T22:58:59.319667+0000 0'0 2024-02-11T22:58:59.319667+0000 0 0 periodic scrub scheduled @ 2024-02-13T09:33:29.063786+0000 0 0 2024-02-11T23:01:13.331 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:01:13.331 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:01:13.331 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:01:13.347 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-02-11T23:01:13.348 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-02-11T23:01:13.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:01:13.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-02-11T23:01:13.389 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:01:13.389 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-02-11T23:01:13.422 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:01:13.423 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-02-11T23:01:13.456 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:01:13.456 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-02-11T23:01:13.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-02-11T23:01:13.489 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-02-11T23:01:13.490 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-02-11T23:01:13.495 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-02-11T22:59:35.749690+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-02-11T22:59:35.749690+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-02-11T22:59:35.749690+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:00:49.040+0000 7f7d78a89640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-02-11T23:01:13.496 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-02-11T23:01:13.497 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-02-11T23:01:13.501 INFO:tasks.workunit.client.0.smithi121.stdout:success 2024-02-11T23:01:13.501 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-02-11T23:01:13.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-02-11T23:01:13.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-02-11T23:01:13.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-02-11T23:01:14.118 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'test' does not exist 2024-02-11T23:01:14.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-02-11T23:01:14.134 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:01:14.134 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:01:14.135 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:01:14.135 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:01:14.135 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:01:14.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:01:14.453 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:01:14.453 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T23:01:14.454 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T23:01:14.454 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T23:01:14.455 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T23:01:14.455 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T23:01:14.456 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T23:01:14.457 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:01:14.457 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T23:01:14.457 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T23:01:14.457 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:01:14.458 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:01:14.459 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-11T23:01:14.459 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-02-11T23:01:14.479 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T23:01:14.479 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:14.479 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:14.479 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26067 2024-02-11T23:01:14.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T23:01:14.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T23:01:14.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-02-11T23:01:14.481 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:01:14.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:01:14.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:01:14.483 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T23:01:14.483 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T23:01:14.484 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T23:01:14.484 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T23:01:14.485 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T23:01:14.485 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T23:01:14.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:01:14.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T23:01:14.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T23:01:14.486 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:01:14.487 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:01:14.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-11T23:01:14.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-02-11T23:01:14.489 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T23:01:14.491 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:14.491 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:14.491 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26067 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:14.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26067 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-02-11T23:01:14.494 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-02-11T23:01:14.495 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-02-11T23:01:14.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-02-11T23:01:14.495 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:01:14.495 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-02-11T23:01:14.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-02-11T23:01:14.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-11T23:01:14.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:01:14.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:01:14.891 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:01:14.891 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:14.891 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:14.891 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:14.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-11T23:01:15.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-11T23:01:15.134 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-11T23:01:15.134 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T23:01:15.134 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T23:01:15.135 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:01:15.136 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:15.136 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:15.136 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-mon.a.asok 2024-02-11T23:01:15.136 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T23:01:15.136 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26067/ceph-mon.a.asok config get fsid 2024-02-11T23:01:15.252 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-11T23:01:15.253 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-11T23:01:15.253 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-11T23:01:15.253 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-11T23:01:15.253 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-11T23:01:15.253 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-11T23:01:15.254 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-11T23:01:15.254 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-11T23:01:15.254 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:01:15.255 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:15.255 INFO:tasks.workunit.client.0.smithi121.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:15.255 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-mon.a.asok 2024-02-11T23:01:15.255 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-11T23:01:15.255 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26067/ceph-mon.a.asok config get mon_host 2024-02-11T23:01:15.365 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-02-11T23:01:15.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-11T23:01:16.281 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-11T23:01:16.281 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:01:16.281 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:01:16.282 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:01:16.282 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:16.282 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:16.282 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:16.282 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T23:01:16.284 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-11T23:01:16.322 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-02-11T23:01:16.322 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T23:01:16.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:01:16.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T23:01:16.325 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:01:16.326 INFO:tasks.workunit.client.0.smithi121.stdout:add osd0 d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:01:16.326 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d8cfa855-aa11-4830-a5ad-a736b4fc4569' 2024-02-11T23:01:16.326 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T23:01:16.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8UcllPV4qFRAARbVtz/PNgrdn6ofRKmrFxg== 2024-02-11T23:01:16.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8UcllPV4qFRAARbVtz/PNgrdn6ofRKmrFxg=="}' 2024-02-11T23:01:16.356 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8cfa855-aa11-4830-a5ad-a736b4fc4569 -i td/divergent-priors/0/new.json 2024-02-11T23:01:16.681 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:01:16.698 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-02-11T23:01:16.699 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8UcllPV4qFRAARbVtz/PNgrdn6ofRKmrFxg== --osd-uuid d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:01:16.740 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.736+0000 7faf18816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:16.758 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.756+0000 7faf18816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:16.760 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.756+0000 7faf18816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:16.761 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.756+0000 7faf18816640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T23:01:16.761 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.756+0000 7faf18816640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T23:01:16.761 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.756+0000 7faf18816640 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-02-11T23:01:16.798 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:16.796+0000 7faf18816640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-02-11T23:01:23.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-02-11T23:01:23.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T23:01:23.080 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd0 key to auth repository 2024-02-11T23:01:23.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-11T23:01:23.080 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T23:01:23.641 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.0 2024-02-11T23:01:23.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-11T23:01:23.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T23:01:23.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:01:23.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T23:01:23.643 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T23:01:23.683 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:23.680+0000 7f9f8ffc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:23.715 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:23.712+0000 7f9f8ffc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:23.716 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:23.712+0000 7f9f8ffc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:24.022 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-11T23:01:24.022 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:01:24.022 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:24.023 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:24.276 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:24.272+0000 7f9f8ffc5640 -1 Falling back to public interface 2024-02-11T23:01:24.448 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:25.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:25.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:25.450 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:01:25.451 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T23:01:25.451 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:25.451 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:25.856 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:26.782 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:26.780+0000 7f9f8ffc5640 -1 osd.0 0 log_to_monitors true 2024-02-11T23:01:26.857 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:26.857 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:26.858 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:01:26.858 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T23:01:26.858 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:26.858 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:27.478 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:28.480 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:28.480 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:28.481 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:01:28.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:01:28.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:28.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:28.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:29.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:29.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:29.905 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T23:01:29.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T23:01:29.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:29.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:30.448 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:31.449 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:31.449 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:31.450 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T23:01:31.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T23:01:31.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:31.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:31.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:32.927 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:32.927 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:32.927 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T23:01:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T23:01:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:33.360 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:34.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:34.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:34.362 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T23:01:34.363 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T23:01:34.363 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:34.363 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:34.803 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:35.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:35.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:35.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T23:01:35.805 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T23:01:35.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:35.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:36.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:37.259 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:37.260 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:37.260 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T23:01:37.260 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T23:01:37.260 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:37.261 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:37.738 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:38.739 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:38.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:38.740 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T23:01:38.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T23:01:38.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:38.740 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:39.256 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:40.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:40.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:40.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T23:01:40.259 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T23:01:40.259 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:40.259 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:41.397 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:42.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:42.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:42.399 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T23:01:42.400 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T23:01:42.400 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:42.400 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:43.188 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:44.190 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:44.190 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:44.190 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T23:01:44.191 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T23:01:44.191 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:44.191 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:44.982 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:45.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:45.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:45.984 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T23:01:45.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T23:01:45.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:45.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:46.720 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:47.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:47.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:47.722 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T23:01:47.723 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T23:01:47.723 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:47.723 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:48.471 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:49.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:49.472 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:49.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T23:01:49.473 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T23:01:49.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:49.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:50.298 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:01:51.300 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:01:51.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:51.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T23:01:51.301 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T23:01:51.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:51.301 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:01:51.744 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1942575785,v1:127.0.0.1:6803/1942575785] [v2:127.0.0.1:6804/1942575785,v1:127.0.0.1:6805/1942575785] exists,up d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-02-11T23:01:51.748 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:01:51.749 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T23:01:51.750 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-02-11T23:01:51.751 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T23:01:51.751 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:01:51.751 INFO:tasks.workunit.client.0.smithi121.stdout:add osd1 e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:01:51.752 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e418b766-1bc3-4cd6-b910-0d1070b7b9e2' 2024-02-11T23:01:51.752 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T23:01:51.779 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfUcllYnBjLhAAcu/MVZxbp/IM2TtQAsu5Jg== 2024-02-11T23:01:51.779 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfUcllYnBjLhAAcu/MVZxbp/IM2TtQAsu5Jg=="}' 2024-02-11T23:01:51.779 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e418b766-1bc3-4cd6-b910-0d1070b7b9e2 -i td/divergent-priors/1/new.json 2024-02-11T23:01:52.312 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:01:52.328 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-02-11T23:01:52.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfUcllYnBjLhAAcu/MVZxbp/IM2TtQAsu5Jg== --osd-uuid e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:01:52.371 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.368+0000 7f634ebbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:52.373 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.372+0000 7f634ebbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:52.374 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.372+0000 7f634ebbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:52.375 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.372+0000 7f634ebbe640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T23:01:52.375 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.372+0000 7f634ebbe640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T23:01:52.375 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.372+0000 7f634ebbe640 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-02-11T23:01:52.412 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:52.408+0000 7f634ebbe640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-02-11T23:01:58.841 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-02-11T23:01:58.841 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T23:01:58.842 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd1 key to auth repository 2024-02-11T23:01:58.842 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-11T23:01:58.842 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T23:01:59.380 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-11T23:01:59.381 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T23:01:59.381 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:01:59.381 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T23:01:59.381 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T23:01:59.381 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T23:01:59.416 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:59.411+0000 7f8fe7fd2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:59.418 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:59.415+0000 7f8fe7fd2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:59.419 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:59.415+0000 7f8fe7fd2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:01:59.800 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:01:59.801 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:01:59.801 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:01:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:01:59.983+0000 7f8fe7fd2640 -1 Falling back to public interface 2024-02-11T23:02:00.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:01.206 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:01.206 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:01.207 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:02:01.207 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T23:02:01.207 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:01.207 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:01.610 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:02.316 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:02.311+0000 7f8fe7fd2640 -1 osd.1 0 log_to_monitors true 2024-02-11T23:02:02.612 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:02.612 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:02.612 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:02:02.613 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T23:02:02.613 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:02.613 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:03.283 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:04.285 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:04.285 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:04.285 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:02:04.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:02:04.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:04.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:04.710 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:05.712 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:05.712 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T23:02:05.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T23:02:05.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:05.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:06.295 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:07.296 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:07.296 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:07.297 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T23:02:07.297 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T23:02:07.297 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:07.297 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:07.792 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:08.794 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:08.794 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:08.795 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T23:02:08.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T23:02:08.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:08.795 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:09.222 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:10.223 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:10.223 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:10.224 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T23:02:10.224 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T23:02:10.224 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:10.224 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:10.693 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:11.694 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:11.694 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:11.695 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T23:02:11.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T23:02:11.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:11.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:12.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:13.177 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:13.177 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:13.177 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T23:02:13.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T23:02:13.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:13.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:13.672 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:14.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:14.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:14.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T23:02:14.675 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T23:02:14.675 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:14.675 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:15.213 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:16.214 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:16.214 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:16.215 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T23:02:16.215 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T23:02:16.215 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:16.215 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:17.189 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:18.191 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:18.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:18.410 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T23:02:18.410 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T23:02:18.411 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:18.411 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:19.380 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:20.382 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:20.382 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:20.382 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T23:02:20.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T23:02:20.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:20.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:21.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:22.250 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:22.251 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:22.251 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T23:02:22.251 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T23:02:22.252 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:22.252 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:23.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:24.182 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:24.182 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:24.182 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T23:02:24.183 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T23:02:24.183 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:24.183 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:25.406 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:26.408 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:26.408 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:26.408 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T23:02:26.408 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T23:02:26.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:26.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:26.936 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:27.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:27.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:27.939 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T23:02:27.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T23:02:27.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:27.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:02:28.411 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2724094408,v1:127.0.0.1:6811/2724094408] [v2:127.0.0.1:6812/2724094408,v1:127.0.0.1:6813/2724094408] exists,up e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:02:28.414 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:02:28.415 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:02:28.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-11T23:02:28.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-02-11T23:02:28.419 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-11T23:02:28.419 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:02:28.419 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f360f5a2-846b-41c8-a82e-66347b28d104' 2024-02-11T23:02:28.419 INFO:tasks.workunit.client.0.smithi121.stdout:add osd2 f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:02:28.419 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-11T23:02:28.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEUsllwnqLGhAAw42LherzbVDT1GRYAjMkYw== 2024-02-11T23:02:28.445 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEUsllwnqLGhAAw42LherzbVDT1GRYAjMkYw=="}' 2024-02-11T23:02:28.446 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f360f5a2-846b-41c8-a82e-66347b28d104 -i td/divergent-priors/2/new.json 2024-02-11T23:02:28.897 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:02:28.912 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-02-11T23:02:28.913 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEUsllwnqLGhAAw42LherzbVDT1GRYAjMkYw== --osd-uuid f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:02:28.954 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:28.951+0000 7fc241e8a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:29.019 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.015+0000 7fc241e8a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:29.020 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.015+0000 7fc241e8a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:29.021 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.019+0000 7fc241e8a640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T23:02:29.021 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.019+0000 7fc241e8a640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T23:02:29.021 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.019+0000 7fc241e8a640 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-02-11T23:02:29.143 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:29.143+0000 7fc241e8a640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-02-11T23:02:35.546 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-02-11T23:02:35.547 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-11T23:02:35.548 INFO:tasks.workunit.client.0.smithi121.stdout:adding osd2 key to auth repository 2024-02-11T23:02:35.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-11T23:02:35.549 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-11T23:02:36.094 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.2 2024-02-11T23:02:36.095 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-11T23:02:36.095 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-11T23:02:36.095 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-11T23:02:36.096 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:02:36.096 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-11T23:02:36.135 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:36.131+0000 7f6d03575640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:36.177 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:36.175+0000 7f6d03575640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:36.179 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:36.175+0000 7f6d03575640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:02:36.484 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-11T23:02:36.485 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:02:36.485 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T23:02:36.485 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:02:36.485 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:02:36.485 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:02:36.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:36.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:02:36.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:36.486 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:36.739 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:36.735+0000 7f6d03575640 -1 Falling back to public interface 2024-02-11T23:02:36.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:37.904 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:37.904 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:37.905 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:02:37.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T23:02:37.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:37.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:38.316 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:39.318 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:39.318 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:39.319 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:02:39.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T23:02:39.319 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:02:39.315+0000 7f6d03575640 -1 osd.2 0 log_to_monitors true 2024-02-11T23:02:39.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:39.319 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:40.086 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:41.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:41.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:41.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:02:41.089 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:02:41.089 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:41.090 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:41.732 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:42.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:42.734 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:42.734 INFO:tasks.workunit.client.0.smithi121.stdout:4 2024-02-11T23:02:42.735 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-11T23:02:42.735 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:42.735 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:43.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:44.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:44.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:44.271 INFO:tasks.workunit.client.0.smithi121.stdout:5 2024-02-11T23:02:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-11T23:02:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:44.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:44.713 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:45.715 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:45.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:45.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-11T23:02:45.716 INFO:tasks.workunit.client.0.smithi121.stdout:6 2024-02-11T23:02:45.716 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:45.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:46.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:47.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:47.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:47.199 INFO:tasks.workunit.client.0.smithi121.stdout:7 2024-02-11T23:02:47.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-11T23:02:47.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:47.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:47.678 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:48.680 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:48.680 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:48.681 INFO:tasks.workunit.client.0.smithi121.stdout:8 2024-02-11T23:02:48.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-11T23:02:48.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:48.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:49.269 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:50.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:50.270 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:50.271 INFO:tasks.workunit.client.0.smithi121.stdout:9 2024-02-11T23:02:50.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-11T23:02:50.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:50.271 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:50.744 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:51.745 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:51.745 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:51.746 INFO:tasks.workunit.client.0.smithi121.stdout:10 2024-02-11T23:02:51.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-11T23:02:51.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:51.746 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:52.248 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:53.250 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:53.250 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:53.250 INFO:tasks.workunit.client.0.smithi121.stdout:11 2024-02-11T23:02:53.250 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-11T23:02:53.251 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:53.251 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:53.975 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:54.977 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:54.977 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:54.977 INFO:tasks.workunit.client.0.smithi121.stdout:12 2024-02-11T23:02:54.978 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-11T23:02:54.978 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:54.978 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:56.009 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:57.010 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:57.011 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:57.011 INFO:tasks.workunit.client.0.smithi121.stdout:13 2024-02-11T23:02:57.011 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-11T23:02:57.012 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:57.012 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:02:58.186 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:02:59.187 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:02:59.187 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:02:59.188 INFO:tasks.workunit.client.0.smithi121.stdout:14 2024-02-11T23:02:59.188 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-11T23:02:59.188 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:02:59.188 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:03:00.149 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:03:01.150 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:03:01.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:03:01.151 INFO:tasks.workunit.client.0.smithi121.stdout:15 2024-02-11T23:03:01.151 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-11T23:03:01.152 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:03:01.152 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:03:02.340 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:03:03.342 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:03:03.342 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:03:03.342 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-11T23:03:03.343 INFO:tasks.workunit.client.0.smithi121.stdout:16 2024-02-11T23:03:03.343 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:03:03.343 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:03:03.952 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:03:04.955 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:03:04.955 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:03:04.956 INFO:tasks.workunit.client.0.smithi121.stdout:17 2024-02-11T23:03:04.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-11T23:03:04.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:03:04.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:03:05.368 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2574987887,v1:127.0.0.1:6819/2574987887] [v2:127.0.0.1:6820/2574987887,v1:127.0.0.1:6821/2574987887] exists,up f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:03:05.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:03:05.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:03:05.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:03:05.369 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-02-11T23:03:06.049 INFO:tasks.workunit.client.0.smithi121.stderr:noout is set 2024-02-11T23:03:06.072 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-02-11T23:03:06.820 INFO:tasks.workunit.client.0.smithi121.stderr:noin is set 2024-02-11T23:03:06.839 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-02-11T23:03:07.665 INFO:tasks.workunit.client.0.smithi121.stderr:nodown is set 2024-02-11T23:03:07.685 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-02-11T23:03:07.685 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-11T23:03:08.271 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'test' already exists 2024-02-11T23:03:08.287 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-11T23:03:09.289 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-02-11T23:03:10.052 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 size to 3 2024-02-11T23:03:10.071 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-02-11T23:03:10.981 INFO:tasks.workunit.client.0.smithi121.stderr:set pool 1 min_size to 2 2024-02-11T23:03:11.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-02-11T23:03:11.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:03:11.007 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:11.488 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:03:11.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-11T23:03:11.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-11T23:03:11.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-11T23:03:11.681 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:11.681 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:03:11.848 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-11T23:03:11.848 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-11T23:03:11.848 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-02-11T23:03:11.848 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:11.848 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:03:12.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-02-11T23:03:12.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-02-11T23:03:12.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509450' 2024-02-11T23:03:12.004 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:12.004 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-11T23:03:12.004 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:12.005 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:03:12.006 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-11T23:03:12.006 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:12.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-11T23:03:12.007 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-11T23:03:12.007 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836504 2024-02-11T23:03:12.007 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:12.425 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836504 2024-02-11T23:03:12.425 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T23:03:13.426 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T23:03:13.427 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:13.854 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-11T23:03:13.854 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:13.855 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-11T23:03:13.855 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:13.857 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T23:03:13.858 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-11T23:03:13.858 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:13.859 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-11T23:03:13.859 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672977 2024-02-11T23:03:13.859 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-11T23:03:13.859 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T23:03:14.435 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-11T23:03:14.435 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:14.436 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-02-11T23:03:14.436 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:14.437 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:03:14.438 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-02-11T23:03:14.438 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:14.439 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-02-11T23:03:14.439 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509450 2024-02-11T23:03:14.440 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-02-11T23:03:14.440 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:03:14.892 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T23:03:14.893 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T23:03:14.894 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T23:03:14.894 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T23:03:14.894 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T23:03:15.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-11T23:03:15.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T23:03:15.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T23:03:15.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T23:03:15.003 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:03:15.004 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:15.465 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:03:15.654 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-02-11T23:03:15.654 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-02-11T23:03:15.654 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-02-11T23:03:15.654 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:15.655 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:03:15.803 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-02-11T23:03:15.803 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-02-11T23:03:15.804 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980' 2024-02-11T23:03:15.804 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:15.804 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:03:15.952 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980 2-64424509453' 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:15.953 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:03:15.954 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-02-11T23:03:15.954 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:15.955 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836506 2024-02-11T23:03:15.955 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-02-11T23:03:15.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-02-11T23:03:15.956 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:16.344 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-02-11T23:03:16.344 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T23:03:17.346 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T23:03:17.347 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:17.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836506 2024-02-11T23:03:17.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:17.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-02-11T23:03:17.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:17.781 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T23:03:17.782 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-02-11T23:03:17.782 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:17.783 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-02-11T23:03:17.783 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-02-11T23:03:17.783 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672980 2024-02-11T23:03:17.783 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T23:03:18.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-02-11T23:03:18.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:18.203 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:18.203 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-11T23:03:18.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:03:18.205 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-11T23:03:18.205 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:18.206 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-11T23:03:18.206 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-11T23:03:18.206 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509453 2024-02-11T23:03:18.207 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:03:18.618 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-11T23:03:18.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T23:03:18.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:03:18.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:03:19.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T23:03:19.143 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:03:19.143 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:03:19.144 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:03:19.144 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:03:19.144 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:03:19.144 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:03:19.144 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:03:19.540 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-11T23:03:19.540 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:03:19.540 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:03:19.540 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:03:20.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-11T23:03:20.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-11T23:03:20.078 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-11T23:03:20.079 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-02-11T23:03:20.079 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-02-11T23:03:20.473 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:03:20.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-02-11T23:03:20.488 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-02-11T23:03:20.488 INFO:tasks.workunit.client.0.smithi121.stdout:primary and soon to be divergent is 1 2024-02-11T23:03:20.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:03:20.887 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:03:20.887 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-11T23:03:09.618518+0000 0'0 28:14 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:18:39.109372+0000 0 0 2024-02-11T23:03:20.887 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:03:20.887 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:03:20.887 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:03:20.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-02-11T23:03:20.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-02-11T23:03:20.905 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-02-11T23:03:20.906 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-02-11T23:03:20.907 INFO:tasks.workunit.client.0.smithi121.stdout:writing initial objects 2024-02-11T23:03:20.907 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-02-11T23:03:20.907 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-02-11T23:03:20.907 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:20.908 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-02-11T23:03:21.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.074 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-02-11T23:03:21.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.233 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-02-11T23:03:21.401 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.401 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-02-11T23:03:21.560 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.560 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-02-11T23:03:21.753 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.753 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-02-11T23:03:21.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:21.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-02-11T23:03:22.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.133 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-02-11T23:03:22.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-02-11T23:03:22.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-02-11T23:03:22.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-02-11T23:03:22.791 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.791 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-02-11T23:03:22.952 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:22.952 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-02-11T23:03:23.118 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:23.118 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-02-11T23:03:23.277 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:23.278 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-02-11T23:03:23.454 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:23.454 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-02-11T23:03:23.663 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:23.663 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-02-11T23:03:23.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:23.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-02-11T23:03:24.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.123 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-02-11T23:03:24.290 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.291 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-02-11T23:03:24.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.450 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-02-11T23:03:24.609 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.609 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-02-11T23:03:24.785 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.785 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-02-11T23:03:24.952 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:24.953 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-02-11T23:03:25.119 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:25.119 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-02-11T23:03:25.304 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:25.304 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-02-11T23:03:25.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:25.497 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-02-11T23:03:25.723 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:25.723 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-02-11T23:03:25.899 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:25.899 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-02-11T23:03:26.049 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.049 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-02-11T23:03:26.209 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.209 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-02-11T23:03:26.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-02-11T23:03:26.552 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.552 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-02-11T23:03:26.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-02-11T23:03:26.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:26.904 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-02-11T23:03:27.071 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.071 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-02-11T23:03:27.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-02-11T23:03:27.398 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.398 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-02-11T23:03:27.567 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.567 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-02-11T23:03:27.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-02-11T23:03:27.884 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:27.885 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-02-11T23:03:28.052 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.052 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-02-11T23:03:28.211 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.211 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-02-11T23:03:28.379 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.379 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-02-11T23:03:28.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.571 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-02-11T23:03:28.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.722 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-02-11T23:03:28.998 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:28.998 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-02-11T23:03:29.174 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:29.174 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-02-11T23:03:29.332 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:29.332 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-02-11T23:03:29.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:29.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-02-11T23:03:29.703 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:29.704 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-02-11T23:03:29.863 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:29.863 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-02-11T23:03:30.014 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.014 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-02-11T23:03:30.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-02-11T23:03:30.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.366 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-02-11T23:03:30.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-02-11T23:03:30.683 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.683 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-02-11T23:03:30.843 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:30.843 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-02-11T23:03:31.027 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.027 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-02-11T23:03:31.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.204 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-02-11T23:03:31.362 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.363 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-02-11T23:03:31.563 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.564 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-02-11T23:03:31.731 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.731 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-02-11T23:03:31.907 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:31.907 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-02-11T23:03:32.058 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.058 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-02-11T23:03:32.217 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.217 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-02-11T23:03:32.368 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.368 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-02-11T23:03:32.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.570 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-02-11T23:03:32.745 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.745 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-02-11T23:03:32.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:32.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-02-11T23:03:33.088 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.088 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-02-11T23:03:33.263 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.263 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-02-11T23:03:33.440 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.440 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-02-11T23:03:33.624 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.625 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-02-11T23:03:33.792 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.792 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-02-11T23:03:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-02-11T23:03:34.118 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.118 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-02-11T23:03:34.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.286 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-02-11T23:03:34.453 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.453 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-02-11T23:03:34.612 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.612 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-02-11T23:03:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-02-11T23:03:34.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:34.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-02-11T23:03:35.106 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.107 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-02-11T23:03:35.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.266 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-02-11T23:03:35.424 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.425 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-02-11T23:03:35.635 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.635 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-02-11T23:03:35.811 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.811 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-02-11T23:03:35.986 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:35.987 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-02-11T23:03:36.154 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.154 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-02-11T23:03:36.321 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.321 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-02-11T23:03:36.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-02-11T23:03:36.656 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.656 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-02-11T23:03:36.815 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.815 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-02-11T23:03:36.975 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:36.975 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-02-11T23:03:37.159 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:37.159 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-02-11T23:03:37.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:37.310 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-02-11T23:03:37.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:37.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-02-11T23:03:37.696 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:37.696 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-02-11T23:03:37.863 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:37.864 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-02-11T23:03:38.030 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-02-11T23:03:38.031 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-02-11T23:03:38.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-02-11T23:03:38.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-02-11T23:03:38.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T23:03:38.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T23:03:38.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T23:03:38.199 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-02-11T23:03:38.200 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T23:03:38.200 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T23:03:38.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T23:03:38.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T23:03:38.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T23:03:38.275 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-02-11T23:03:38.275 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T23:03:38.275 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T23:03:38.276 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T23:03:38.276 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:03:38.276 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:03:38.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:03:38.701 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:03:38.701 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:03:38.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:03:38.702 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:38.702 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:03:38.872 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-02-11T23:03:38.872 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-02-11T23:03:38.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-02-11T23:03:38.873 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:38.873 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:03:39.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-11T23:03:39.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-11T23:03:39.034 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672987' 2024-02-11T23:03:39.034 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:03:39.034 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672987 2-64424509459' 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-02-11T23:03:39.197 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:39.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:03:39.199 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-02-11T23:03:39.199 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:39.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-02-11T23:03:39.200 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 21474836513 2024-02-11T23:03:39.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-02-11T23:03:39.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:39.605 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836513 2024-02-11T23:03:39.605 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T23:03:40.607 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T23:03:40.607 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:03:41.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-02-11T23:03:41.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:41.045 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-11T23:03:41.045 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:41.046 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-11T23:03:41.047 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-11T23:03:41.047 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:41.048 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-11T23:03:41.048 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-11T23:03:41.049 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.1 seq 42949672987 2024-02-11T23:03:41.049 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-11T23:03:41.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-02-11T23:03:41.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:03:41.496 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-11T23:03:41.496 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:03:41.498 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:03:41.498 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-11T23:03:41.498 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:03:41.500 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-11T23:03:41.500 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-11T23:03:41.500 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 64424509459 2024-02-11T23:03:41.500 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:03:41.889 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-02-11T23:03:41.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T23:03:41.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:03:41.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:03:42.455 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:03:42.456 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:03:42.456 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:03:42.898 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-11T23:03:42.899 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:03:42.900 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:03:42.900 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:03:43.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-11T23:03:43.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-11T23:03:43.409 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-11T23:03:43.410 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-02-11T23:03:43.410 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-02-11T23:03:43.410 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-02-11T23:03:43.410 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-02-11T23:03:43.410 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-11T23:03:43.821 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-02-11T23:03:43.821 INFO:tasks.workunit.client.0.smithi121.stdout:blackholing osds 0 2 2024-02-11T23:03:43.821 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-02-11T23:03:43.821 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:03:44.239 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:03:44.239 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-02-11T23:03:09.618518+0000 28'100 28:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:18:39.109372+0000 0 0 2024-02-11T23:03:44.239 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:03:44.239 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:03:44.240 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:03:44.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:03:44.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-02-11T23:03:44.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-11T23:03:44.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-11T23:03:44.257 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:03:44.257 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:03:44.257 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:03:44.257 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.0.asok 2024-02-11T23:03:44.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:03:44.258 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.0.asok config set objectstore_blackhole 1 2024-02-11T23:03:44.361 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:03:44.362 INFO:tasks.workunit.client.0.smithi121.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-02-11T23:03:44.362 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:03:44.374 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:03:44.375 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-02-11T23:03:44.375 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-02-11T23:03:44.375 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-02-11T23:03:44.376 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:03:44.376 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:03:44.376 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:03:44.377 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.2.asok 2024-02-11T23:03:44.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:03:44.377 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.2.asok config set objectstore_blackhole 1 2024-02-11T23:03:44.481 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:03:44.481 INFO:tasks.workunit.client.0.smithi121.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-02-11T23:03:44.481 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:03:44.492 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-02-11T23:03:44.492 INFO:tasks.workunit.client.0.smithi121.stdout:create new divergent objects 2024-02-11T23:03:44.493 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-02-11T23:03:44.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-02-11T23:03:44.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-02-11T23:03:44.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-02-11T23:03:44.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-02-11T23:03:44.495 INFO:tasks.workunit.client.0.smithi121.stdout:writing divergent objects 2024-02-11T23:03:44.495 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-02-11T23:03:44.496 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-02-11T23:03:44.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-02-11T23:03:44.496 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-02-11T23:03:44.497 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-02-11T23:03:44.497 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-02-11T23:03:44.497 INFO:tasks.workunit.client.0.smithi121.stdout:remove divergent objects 2024-02-11T23:03:44.498 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-02-11T23:03:44.498 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-02-11T23:03:44.499 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-02-11T23:03:44.500 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-02-11T23:03:44.501 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-02-11T23:03:44.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-02-11T23:03:44.502 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-02-11T23:03:44.502 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-02-11T23:03:44.503 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-02-11T23:03:44.503 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-02-11T23:03:44.504 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-02-11T23:03:54.505 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-02-11T23:03:54.511 INFO:tasks.workunit.client.0.smithi121.stderr:rados: no process found 2024-02-11T23:03:54.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-02-11T23:03:54.512 INFO:tasks.workunit.client.0.smithi121.stdout:killing all the osds 2024-02-11T23:03:54.512 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:03:54.900 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:03:54.900 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-02-11T23:03:09.618518+0000 28'106 28:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:18:39.109372+0000 0 0 2024-02-11T23:03:54.900 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:03:54.900 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:03:54.901 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:03:54.916 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-02-11T23:03:54.916 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:03:54.916 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:03:54.917 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:03:54.917 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:03:54.917 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:03:55.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:03:55.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-02-11T23:03:55.025 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-02-11T23:03:55.530 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already down. 2024-02-11T23:03:55.547 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-02-11T23:03:55.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-02-11T23:03:56.157 INFO:tasks.workunit.client.0.smithi121.stderr:osd.1 is already down. 2024-02-11T23:03:56.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-02-11T23:03:56.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-02-11T23:03:56.733 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already down. 2024-02-11T23:03:56.751 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:03:56.751 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-02-11T23:03:57.307 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already out. 2024-02-11T23:03:57.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:03:57.324 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-02-11T23:03:57.909 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already out. 2024-02-11T23:03:57.926 INFO:tasks.workunit.client.0.smithi121.stdout:bringing up non_divergent 0 2 2024-02-11T23:03:57.926 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-02-11T23:03:57.926 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:03:58.312 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:03:58.312 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-02-11T23:03:09.618518+0000 28'106 28:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T03:18:39.109372+0000 0 0 2024-02-11T23:03:58.312 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:03:58.313 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:03:58.313 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:03:58.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:03:58.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-02-11T23:03:58.329 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:03:58.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:03:58.331 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:03:58.331 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:03:58.331 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:03:58.332 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:03:58.332 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:03:58.333 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-02-11T23:03:58.334 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.0 2024-02-11T23:03:58.334 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:03:58.335 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:03:58.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:03:58.374+0000 7efc71004640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:03:58.381 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:03:58.378+0000 7efc71004640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:03:58.383 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:03:58.378+0000 7efc71004640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:03:58.717 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:03:58.718 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:03:58.719 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:03:58.935 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:03:58.930+0000 7efc71004640 -1 Falling back to public interface 2024-02-11T23:03:59.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:00.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:00.201 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:00.202 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:04:00.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T23:04:00.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:00.202 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:04:00.624 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:01.625 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:01.625 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:01.626 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:04:01.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T23:04:01.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:01.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:04:02.045 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:02.113 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:02.110+0000 7efc71004640 -1 osd.0 28 log_to_monitors true 2024-02-11T23:04:03.046 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:03.046 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:03.047 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:04:03.047 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:04:03.047 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:03.047 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:04:03.521 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2610045605,v1:127.0.0.1:6803/2610045605] [v2:127.0.0.1:6804/2610045605,v1:127.0.0.1:6805/2610045605] exists,up d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:04:03.523 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:04:03.524 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:04:03.525 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:04:03.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:04:03.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:04:03.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:04:03.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:04:03.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-02-11T23:04:03.527 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.2 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:04:03.528 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:04:03.529 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:04:03.566 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:03.562+0000 7ff21a5fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:04:03.583 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:03.578+0000 7ff21a5fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:04:03.584 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:03.578+0000 7ff21a5fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:04:03.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-11T23:04:03.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:04:03.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T23:04:03.934 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:03.935 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:04:04.126 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:04.122+0000 7ff21a5fc640 -1 Falling back to public interface 2024-02-11T23:04:04.339 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:05.340 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:05.341 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:05.341 INFO:tasks.workunit.client.0.smithi121.stdout:1 2024-02-11T23:04:05.341 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-11T23:04:05.341 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:05.342 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:04:05.814 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:06.816 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:06.816 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:06.816 INFO:tasks.workunit.client.0.smithi121.stdout:2 2024-02-11T23:04:06.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-11T23:04:06.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:06.817 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:04:06.901 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:04:06.898+0000 7ff21a5fc640 -1 osd.2 28 log_to_monitors true 2024-02-11T23:04:07.228 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-11T23:04:08.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-11T23:04:08.230 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:04:08.230 INFO:tasks.workunit.client.0.smithi121.stdout:3 2024-02-11T23:04:08.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-11T23:04:08.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:04:08.231 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:04:08.700 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3897082287,v1:127.0.0.1:6811/3897082287] [v2:127.0.0.1:6812/3897082287,v1:127.0.0.1:6813/3897082287] exists,up f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:04:08.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:04:08.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:04:08.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:04:08.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:04:08.701 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-02-11T23:04:09.233 INFO:tasks.workunit.client.0.smithi121.stderr:osd.0 is already in. 2024-02-11T23:04:09.248 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:04:09.249 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-02-11T23:04:09.804 INFO:tasks.workunit.client.0.smithi121.stderr:osd.2 is already in. 2024-02-11T23:04:09.821 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-02-11T23:04:09.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-02-11T23:04:09.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T23:04:09.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T23:04:09.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T23:04:09.822 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-02-11T23:04:09.823 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T23:04:09.823 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T23:04:09.823 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T23:04:09.824 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T23:04:09.824 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T23:04:09.889 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-02-11T23:04:09.890 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T23:04:09.890 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T23:04:09.890 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T23:04:09.890 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:04:09.890 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:10.349 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:04:10.548 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-02-11T23:04:10.549 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-02-11T23:04:10.549 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364' 2024-02-11T23:04:10.549 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:10.549 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:04:10.665 INFO:tasks.workunit.client.0.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-11T23:04:10.667 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-02-11T23:04:10.667 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-02-11T23:04:10.667 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-02-11T23:04:10.667 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:10.667 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:04:10.822 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789955 2024-02-11T23:04:10.823 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789955 2024-02-11T23:04:10.823 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2024-02-11T23:04:10.823 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:04:10.823 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855364 2024-02-11T23:04:10.823 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:04:10.825 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:04:10.826 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855364 2024-02-11T23:04:10.826 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:04:10.827 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-02-11T23:04:10.827 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 150323855364 2024-02-11T23:04:10.827 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-02-11T23:04:10.827 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:04:11.236 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855364 2024-02-11T23:04:11.236 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:04:11.237 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789955 2024-02-11T23:04:11.237 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:04:11.237 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:04:11.238 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789955 2024-02-11T23:04:11.238 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:04:11.239 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789955 2024-02-11T23:04:11.240 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 158913789955 2024-02-11T23:04:11.240 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-02-11T23:04:11.240 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:04:11.690 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789953 -lt 158913789955 2024-02-11T23:04:11.691 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-11T23:04:12.692 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-11T23:04:12.693 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:04:13.145 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789955 -lt 158913789955 2024-02-11T23:04:13.145 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T23:04:13.146 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:13.146 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:13.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T23:04:13.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:13.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:14.122 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:14.122 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:14.123 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:14.123 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:14.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:14.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-11T23:04:14.677 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-11T23:04:14.678 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-11T23:04:14.678 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:14.678 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-11T23:04:14.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:14.778 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:14.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:14.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:14.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:14.779 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:14.780 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:14.780 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:15.191 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:15.191 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:15.192 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:15.192 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:15.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:15.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:15.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:15.697 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:15.698 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:15.698 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:15.698 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:15.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:15.698 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:16.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:16.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:16.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-02-11T23:04:16.226 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:16.227 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-11T23:04:16.427 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:16.427 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:16.428 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:16.428 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:16.428 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:16.428 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:16.429 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:16.429 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:16.814 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:16.815 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:16.815 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:16.815 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:17.350 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:17.350 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:17.351 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:17.887 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:17.887 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:17.887 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-02-11T23:04:17.887 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:17.887 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-11T23:04:18.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:18.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:18.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:18.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:18.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:18.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:18.290 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:18.290 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:18.688 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:18.689 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:18.689 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:18.689 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:19.205 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:19.206 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:19.206 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:19.724 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:19.724 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:19.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-02-11T23:04:19.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:19.725 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-11T23:04:20.525 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:20.526 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:20.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:20.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:20.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:20.526 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:20.527 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:20.527 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:20.939 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:20.939 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:20.940 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:20.940 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:21.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:21.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:21.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:21.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:21.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:21.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:21.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:21.464 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:21.464 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:21.983 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:21.983 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:21.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-02-11T23:04:21.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:21.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-02-11T23:04:23.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:23.585 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:23.585 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:23.585 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:23.586 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:23.586 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:23.586 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:23.586 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:23.990 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:23.990 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:23.990 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:23.991 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:24.510 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:24.510 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:24.510 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:24.511 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:25.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:25.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:25.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-02-11T23:04:25.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:25.087 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-02-11T23:04:28.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:28.288 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:28.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:28.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:28.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:28.289 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:28.290 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:28.290 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:28.705 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:28.706 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:28.706 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:28.706 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:29.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:29.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:29.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:29.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:29.256 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:29.256 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:29.256 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:29.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:29.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:29.808 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:29.808 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:29.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-02-11T23:04:29.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:29.809 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-02-11T23:04:36.209 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:36.210 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:36.210 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:36.211 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:36.211 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:36.211 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:36.211 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:36.211 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:36.624 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:36.624 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:36.625 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:36.625 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:37.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:37.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:37.181 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:37.181 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:37.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:37.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:37.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-02-11T23:04:37.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:37.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-02-11T23:04:44.996 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:44.996 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:44.997 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:44.997 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:44.997 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:44.997 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:44.998 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:44.998 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:45.422 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:45.422 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:45.423 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:45.423 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:45.969 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:45.969 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:45.969 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:45.969 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:45.970 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:45.970 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:45.970 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:45.970 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:45.970 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:46.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:46.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:46.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-02-11T23:04:46.482 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-02-11T23:04:47.021 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:04:47.021 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_fingerprint": "5fc69571-41ff-4e98-874b-7575aa9aed0e", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "commit": "d4ed20cfb238ef2c47d64ca4533eca00b8968b98", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "timestamp": "2024-02-11T23:04:47.019930+0000", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "tag": "", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "health": { 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "status": "HEALTH_WARN", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "checks": { 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "OSDMAP_FLAGS": { 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "nodown,noout,noin flag(s) set", 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 17 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [], 2024-02-11T23:04:47.022 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "OSD_DOWN": { 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 osds down", 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "osd.1 (root=default,host=smithi121) is down" 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.023 INFO:tasks.workunit.client.0.smithi121.stdout: "PG_DEGRADED": { 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "POOL_APP_NOT_ENABLED": { 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:04:47.024 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 pool(s) do not have an application enabled", 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "application not enabled on pool 'test'" 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.025 INFO:tasks.workunit.client.0.smithi121.stdout: "mutes": [] 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_first_committed": 1, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_last_committed": 1, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap": { 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:01:14.519955Z", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T23:01:14.519955Z", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release": 19, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release_name": "squid", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "election_strategy": 1, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "disallowed_leaders: ": "", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": false, 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "tiebreaker_mon": "", 2024-02-11T23:04:47.026 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_ranks: ": "", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "features": { 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "persistent": [ 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "kraken", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "luminous", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "mimic", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap-prune", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "nautilus", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "octopus", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "pacific", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "elector-pinging", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "quincy", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "reef", 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "squid" 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.027 INFO:tasks.workunit.client.0.smithi121.stdout: "optional": [] 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "mons": [ 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "rank": 0, 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "a", 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115", 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 0 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.028 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115/0", 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:7115/0", 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "priority": 0, 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 0, 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_location": "{}" 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "quorum": [ 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: 0 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap": { 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 40, 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T23:01:15.436470+0000", 2024-02-11T23:04:47.029 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:04:10.083801+0000", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "last_up_change": "2024-02-11T23:04:07.305282+0000", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "last_in_change": "2024-02-11T23:04:09.636167+0000", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_num": 5800384, 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_set": [ 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "nodown", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "noin", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "noout", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "pglog_hardlimit", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snapdirs", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "recovery_deletes", 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "sortbitwise" 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.030 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_version": 9, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "full_ratio": 0.99000000953674316, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_snapshot": "", 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_max": 1, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "max_osd": 3, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "min_compat_client": "jewel", 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "require_osd_release": "squid", 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "allow_crimson": false, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "pools": [ 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_name": "test", 2024-02-11T23:04:47.031 INFO:tasks.workunit.client.0.smithi121.stdout: "create_time": "2024-02-11T23:03:08.063397+0000", 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": 1, 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_names": "hashpspool", 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "size": 3, 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "min_size": 2, 2024-02-11T23:04:47.032 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_rule": 0, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "object_hash": 2, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num": 1, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num": 1, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num_target": 1, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_target": 1, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_pending": 1, 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pg_merge_meta": { 2024-02-11T23:04:47.035 INFO:tasks.workunit.client.0.smithi121.stdout: "source_pgid": "0.0", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "ready_epoch": 0, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_started": 0, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": 0, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "source_version": "0'0", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "target_version": "0'0" 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_change": "28", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend": "0", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "auid": 0, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_seq": 0, 2024-02-11T23:04:47.036 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_epoch": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_snaps": [], 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps": "[]", 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_bytes": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_objects": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "tiers": [], 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "tier_of": -1, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "read_tier": -1, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "write_tier": -1, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_mode": "none", 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_bytes": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_objects": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-11T23:04:47.037 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profile": "", 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_params": { 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "none" 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_period": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_count": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "use_gmt_hitset": true, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "grade_table": [], 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "stripe_width": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "expected_num_objects": 0, 2024-02-11T23:04:47.038 INFO:tasks.workunit.client.0.smithi121.stdout: "fast_read": false, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "options": {}, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "application_metadata": {}, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "read_balance": { 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "score_acting": 2.9900000095367432, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "score_stable": 2.9900000095367432, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_score": 0.67000001668930054, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_acting": 2, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_stable": 2, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity_weighted": 1, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity": 1, 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity_weighted": 1 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: "osds": [ 2024-02-11T23:04:47.039 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "d8cfa855-aa11-4830-a5ad-a736b4fc4569", 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 5, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 28, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 35, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 39, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 29, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:04:47.040 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6802", 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6803", 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.041 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6804", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6805", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6808", 2024-02-11T23:04:47.042 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6809", 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6806", 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.043 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6807", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6803/2610045605", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6805/2610045605", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2610045605", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2610045605", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:04:47.044 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "e418b766-1bc3-4cd6-b910-0d1070b7b9e2", 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 0, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 0, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 0, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 10, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 23, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 30, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:04:47.045 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.046 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.047 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:04:47.048 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/2724094408", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/2724094408", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2724094408", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2724094408", 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: "exists" 2024-02-11T23:04:47.049 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "f360f5a2-846b-41c8-a82e-66347b28d104", 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 15, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 28, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 37, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 0, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 31, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:04:47.050 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.051 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.052 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:04:47.053 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/3897082287", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/3897082287", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3897082287", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3897082287", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:04:47.054 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_xinfo": [ 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:55.355411+0000", 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:01:50.296898+0000", 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.055 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:55.944413+0000", 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:02:26.626900+0000", 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:56.525347+0000", 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:04:47.056 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:03:03.950310+0000", 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap": [], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_items": [], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_primaries": [], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_temp": [], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_temp": [], 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "blocklist": {}, 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "range_blocklist": {}, 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profiles": { 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "default": { 2024-02-11T23:04:47.057 INFO:tasks.workunit.client.0.smithi121.stdout: "k": "2", 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "m": "2", 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "plugin": "jerasure", 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "technique": "reed_sol_van" 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps_queue": [], 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "new_removed_snaps": [], 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "new_purged_snaps": [], 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_node_flags": {}, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "device_class_flags": {}, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": { 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-11T23:04:47.058 INFO:tasks.workunit.client.0.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_metadata": [ 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6804/2610045605,v1:127.0.0.1:6805/2610045605]", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:04:47.059 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:01:22.387769Z", 2024-02-11T23:04:47.060 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6802/2610045605,v1:127.0.0.1:6803/2610045605]", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2610045605,v1:127.0.0.1:6809/2610045605]", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2610045605,v1:127.0.0.1:6807/2610045605]", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:04:47.061 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/0", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/2724094408,v1:127.0.0.1:6813/2724094408]", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:04:47.062 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:04:47.063 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:01:58.183936Z", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/2724094408,v1:127.0.0.1:6811/2724094408]", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2724094408,v1:127.0.0.1:6817/2724094408]", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2724094408,v1:127.0.0.1:6815/2724094408]", 2024-02-11T23:04:47.064 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/1", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.065 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/3897082287,v1:127.0.0.1:6813/3897082287]", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:04:47.066 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:02:34.824016Z", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:04:47.067 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/3897082287,v1:127.0.0.1:6811/3897082287]", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3897082287,v1:127.0.0.1:6817/3897082287]", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3897082287,v1:127.0.0.1:6815/3897082287]", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:04:47.068 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/2", 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_clean_epochs": { 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "min_last_epoch_clean": 0, 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": { 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "per_pool": [] 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_epochs": [ 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 35 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.069 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 10 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 37 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_first_committed": 1, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_last_committed": 40, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "crushmap": { 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": [ 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:04:47.070 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.0", 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.1", 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.2", 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: "types": [ 2024-02-11T23:04:47.071 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 0, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd" 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "host" 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 2, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "chassis" 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 3, 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "rack" 2024-02-11T23:04:47.072 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 4, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "row" 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 5, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pdu" 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 6, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pod" 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 7, 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "room" 2024-02-11T23:04:47.073 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 8, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "datacenter" 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 9, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "zone" 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 10, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "region" 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "root" 2024-02-11T23:04:47.074 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "buckets": [ 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -1, 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default", 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:04:47.075 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -2, 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default~hdd", 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:04:47.076 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121", 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:04:47.077 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.078 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121~hdd", 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:04:47.079 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "rules": [ 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_id": 0, 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_name": "replicated_rule", 2024-02-11T23:04:47.080 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "steps": [ 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "take", 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "item": -1, 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "item_name": "default" 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "choose_firstn", 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 0, 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "osd" 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "emit" 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:04:47.081 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "tunables": { 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_tries": 0, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_fallback_tries": 0, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_total_tries": 50, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_descend_once": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_vary_r": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_stable": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "straw_calc_version": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "allowed_bucket_algs": 54, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "profile": "jewel", 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_tunables": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "legacy_tunables": 0, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "minimum_required_version": "jewel", 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables": 1, 2024-02-11T23:04:47.082 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables2": 1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v2_rules": 0, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables3": 1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v3_rules": 0, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v4_buckets": 1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables5": 1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v5_rules": 0 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_args": {} 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "fsmap": { 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "default_fscid": -1, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": { 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": {}, 2024-02-11T23:04:47.083 INFO:tasks.workunit.client.0.smithi121.stdout: "ro_compat": {}, 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "incompat": { 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_1": "base v0.20", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_2": "client writeable ranges", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_3": "default file layouts on dirs", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_4": "dir inode in separate object", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_5": "mds uses versioned encoding", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_6": "dirfrag is stored in omap", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_8": "no anchor table", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_9": "file layout v2", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_10": "snaprealm v2", 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_11": "minor log segments" 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_flags": { 2024-02-11T23:04:47.084 INFO:tasks.workunit.client.0.smithi121.stdout: "enable_multiple": true, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "ever_enabled_multiple": true 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "standbys": [], 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "filesystems": [] 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_first_committed": 1, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_last_committed": 1, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "auth": { 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 5, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "num_secrets": 3 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_active": 1, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_unknown": 0, 2024-02-11T23:04:47.085 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osd": 3, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_sum": { 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 61700, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:04:47.086 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 100, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 100, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:04:47.087 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:04:47.088 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 16, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 16, 2024-02-11T23:04:47.089 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 0 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum": { 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83596, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2080, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:04:47.090 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489204, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036944896, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2129920, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 450286, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10397, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416931 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:04:47.091 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 0, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum_by_class": { 2024-02-11T23:04:47.092 INFO:tasks.workunit.client.0.smithi121.stdout: "hdd": { 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83596, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2080, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489204, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:04:47.093 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036944896, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2129920, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 450286, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10397, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416931 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 0, 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:04:47.094 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_stats": [ 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "poolid": 1, 2024-02-11T23:04:47.095 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 61700, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 100, 2024-02-11T23:04:47.096 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 100, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:04:47.097 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:04:47.098 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 1220608, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 184566, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 16, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 16, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 3 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.099 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_stats": [ 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 150323855371 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 42949672990 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 158913789962 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_state": [ 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.100 INFO:tasks.workunit.client.0.smithi121.stdout: "state": "active+undersized+degraded", 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 1 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_osd": [ 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 1, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:04:47.101 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 0, 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": [ 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": {} 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "servicemap": { 2024-02-11T23:04:47.102 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 5, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:03:58.760282+0000", 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "services": {} 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_first_committed": 1, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_last_committed": 95, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_first_committed": 1, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_last_committed": 384, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "paxos": { 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 562, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pn": 0, 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: "accepted_pn": 0 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:04:47.103 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:04:47.104 INFO:tasks.workunit.client.0.smithi121.stderr:report 3071146937 2024-02-11T23:04:47.104 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-02-11T23:04:47.104 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-02-11T23:04:47.104 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-02-11T23:04:47.105 INFO:tasks.workunit.client.0.smithi121.stdout:writing non-divergent object existing_4 2024-02-11T23:04:47.105 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-02-11T23:04:47.105 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:04:47.416 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:04:47.417 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 28'100 40:121 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:04:47.417 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:04:47.417 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:04:47.417 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:04:47.432 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-02-11T23:04:47.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-02-11T23:04:47.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-02-11T23:04:47.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-11T23:04:47.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-11T23:04:47.597 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-11T23:04:47.598 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-02-11T23:04:47.598 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-11T23:04:47.598 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-11T23:04:47.598 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-11T23:04:47.599 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-11T23:04:47.599 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-11T23:04:47.669 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-02-11T23:04:47.669 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-11T23:04:47.669 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-11T23:04:47.670 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-11T23:04:47.670 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-11T23:04:47.670 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-11T23:04:48.092 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-11T23:04:48.092 INFO:tasks.workunit.client.0.smithi121.stderr:1 2024-02-11T23:04:48.092 INFO:tasks.workunit.client.0.smithi121.stderr:2' 2024-02-11T23:04:48.093 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-11T23:04:48.093 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:48.093 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-11T23:04:48.287 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855373 2024-02-11T23:04:48.287 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855373 2024-02-11T23:04:48.287 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855373' 2024-02-11T23:04:48.287 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:48.287 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-11T23:04:48.414 INFO:tasks.workunit.client.0.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-11T23:04:48.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-02-11T23:04:48.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-02-11T23:04:48.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-02-11T23:04:48.417 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-11T23:04:48.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-11T23:04:48.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789965 2024-02-11T23:04:48.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789965 2024-02-11T23:04:48.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855373 2-158913789965' 2024-02-11T23:04:48.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:04:48.580 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855373 2024-02-11T23:04:48.580 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:04:48.580 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-11T23:04:48.581 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855373 2024-02-11T23:04:48.581 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:04:48.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855373 2024-02-11T23:04:48.582 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2024-02-11T23:04:48.582 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.0 seq 150323855373 2024-02-11T23:04:48.583 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-11T23:04:48.989 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855374 -lt 150323855373 2024-02-11T23:04:48.989 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-11T23:04:48.990 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789965 2024-02-11T23:04:48.990 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-11T23:04:48.991 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-11T23:04:48.992 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789965 2024-02-11T23:04:48.992 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-11T23:04:48.993 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789965 2024-02-11T23:04:48.993 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789965' 2024-02-11T23:04:48.994 INFO:tasks.workunit.client.0.smithi121.stdout:waiting osd.2 seq 158913789965 2024-02-11T23:04:48.994 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-11T23:04:49.390 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789965 -lt 158913789965 2024-02-11T23:04:49.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-11T23:04:49.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:49.391 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:49.900 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-11T23:04:49.903 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:49.904 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:49.904 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:49.904 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:49.904 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:49.905 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:49.905 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:50.352 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:50.353 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:50.353 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:50.353 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:50.925 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-11T23:04:51.026 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:51.026 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:51.027 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:51.027 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:51.027 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:51.027 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:51.027 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:51.028 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:51.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:51.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:51.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:51.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:51.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:51.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:51.937 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:51.938 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:52.507 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:52.507 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:52.507 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-02-11T23:04:52.507 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:52.508 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-11T23:04:52.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:52.708 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:52.709 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:52.709 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:52.709 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:52.710 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:52.710 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:52.710 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:53.112 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:53.112 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:53.113 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:53.113 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:53.621 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:53.622 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:53.622 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:54.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:54.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:54.157 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-02-11T23:04:54.158 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:54.158 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-11T23:04:54.558 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:54.558 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:54.559 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:54.559 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:54.559 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:54.559 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:54.560 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:54.560 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:54.954 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:54.954 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:54.955 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:54.955 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:55.473 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:55.474 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:55.474 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:55.474 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:56.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:56.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:56.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-02-11T23:04:56.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:56.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-11T23:04:56.842 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:04:56.842 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:04:56.843 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:04:57.255 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:04:57.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:04:57.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:04:57.256 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:04:57.805 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:04:57.806 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:04:58.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:04:58.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:04:58.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-02-11T23:04:58.399 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:04:58.400 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-02-11T23:05:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:05:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:05:00.002 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:05:00.002 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:05:00.002 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:05:00.002 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:05:00.003 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:05:00.003 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:05:00.416 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:05:00.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:05:00.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:05:00.417 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:05:00.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:05:00.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:05:00.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:05:00.956 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:05:00.957 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:05:00.957 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:05:00.957 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:05:00.957 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:05:00.957 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:05:01.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:05:01.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:05:01.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-02-11T23:05:01.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:05:01.494 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-02-11T23:05:04.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:05:04.695 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:05:04.695 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:05:04.696 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:05:04.696 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:05:04.696 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:05:04.696 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:05:04.696 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:05:05.107 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:05:05.107 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:05:05.108 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:05:05.108 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:05:05.619 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:05:05.620 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:05:06.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:05:06.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:05:06.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-02-11T23:05:06.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:05:06.139 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-02-11T23:05:12.540 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:05:12.541 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:05:12.541 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:05:12.541 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:05:12.541 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:05:12.541 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:05:12.542 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:05:12.542 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:05:12.962 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:05:12.963 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:05:12.963 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:05:12.963 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:05:13.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:05:13.494 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:05:13.494 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:05:14.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:05:14.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:05:14.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-02-11T23:05:14.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-11T23:05:14.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-02-11T23:05:21.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-11T23:05:21.355 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-11T23:05:21.356 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-11T23:05:21.356 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-11T23:05:21.356 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-11T23:05:21.356 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-11T23:05:21.357 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-11T23:05:21.357 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-11T23:05:21.938 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-11T23:05:21.939 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-11T23:05:21.939 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-11T23:05:21.940 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-11T23:05:22.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-11T23:05:22.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-11T23:05:22.469 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-11T23:05:22.470 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-11T23:05:23.005 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-11T23:05:23.005 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-11T23:05:23.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-02-11T23:05:23.006 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-02-11T23:05:23.535 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_fingerprint": "5fc69571-41ff-4e98-874b-7575aa9aed0e", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "commit": "d4ed20cfb238ef2c47d64ca4533eca00b8968b98", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "timestamp": "2024-02-11T23:05:23.534203+0000", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "tag": "", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "health": { 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "status": "HEALTH_WARN", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "checks": { 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "OSDMAP_FLAGS": { 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:05:23.536 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "nodown,noout,noin flag(s) set", 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 17 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [], 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "OSD_DOWN": { 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 osds down", 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:05:23.537 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "osd.1 (root=default,host=smithi121) is down" 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "PG_DEGRADED": { 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 2 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:05:23.538 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "POOL_APP_NOT_ENABLED": { 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "severity": "HEALTH_WARN", 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "summary": { 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "1 pool(s) do not have an application enabled", 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "count": 1 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: "detail": [ 2024-02-11T23:05:23.539 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "application not enabled on pool 'test'" 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: "muted": false 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: "mutes": [] 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.540 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_first_committed": 1, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap_last_committed": 1, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "monmap": { 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:01:14.519955Z", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T23:01:14.519955Z", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release": 19, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "min_mon_release_name": "squid", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "election_strategy": 1, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "disallowed_leaders: ": "", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": false, 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "tiebreaker_mon": "", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_ranks: ": "", 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "features": { 2024-02-11T23:05:23.541 INFO:tasks.workunit.client.0.smithi121.stdout: "persistent": [ 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "kraken", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "luminous", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "mimic", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap-prune", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "nautilus", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "octopus", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "pacific", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "elector-pinging", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "quincy", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "reef", 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "squid" 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: "optional": [] 2024-02-11T23:05:23.542 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "mons": [ 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "rank": 0, 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "a", 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115", 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 0 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.543 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:7115/0", 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:7115/0", 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "priority": 0, 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 0, 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_location": "{}" 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "quorum": [ 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: 0 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap": { 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 40, 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "fsid": "c15e6ddc-69cf-4a83-aaf1-df560b419426", 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "created": "2024-02-11T23:01:15.436470+0000", 2024-02-11T23:05:23.544 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:04:10.083801+0000", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "last_up_change": "2024-02-11T23:04:07.305282+0000", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "last_in_change": "2024-02-11T23:04:09.636167+0000", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_num": 5800384, 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_set": [ 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "nodown", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "noin", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "noout", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "pglog_hardlimit", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snapdirs", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "recovery_deletes", 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "sortbitwise" 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.545 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_version": 9, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "full_ratio": 0.99000000953674316, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_snapshot": "", 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_max": 1, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "max_osd": 3, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "min_compat_client": "jewel", 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "require_osd_release": "squid", 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "allow_crimson": false, 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: "pools": [ 2024-02-11T23:05:23.546 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_name": "test", 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "create_time": "2024-02-11T23:03:08.063397+0000", 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "flags": 1, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "flags_names": "hashpspool", 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "size": 3, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "min_size": 2, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_rule": 0, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-11T23:05:23.547 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "object_hash": 2, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num": 1, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num": 1, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_placement_num_target": 1, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_target": 1, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_num_pending": 1, 2024-02-11T23:05:23.548 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pg_merge_meta": { 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "source_pgid": "0.0", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "ready_epoch": 0, 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_started": 0, 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": 0, 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "source_version": "0'0", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "target_version": "0'0" 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_change": "28", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend": "0", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "auid": 0, 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-11T23:05:23.549 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_seq": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_epoch": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_snaps": [], 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps": "[]", 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_bytes": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "quota_max_objects": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "tiers": [], 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "tier_of": -1, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "read_tier": -1, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "write_tier": -1, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_mode": "none", 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_bytes": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "target_max_objects": 0, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-11T23:05:23.550 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profile": "", 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_params": { 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "none" 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_period": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_count": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "use_gmt_hitset": true, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "grade_table": [], 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "stripe_width": 0, 2024-02-11T23:05:23.551 INFO:tasks.workunit.client.0.smithi121.stdout: "expected_num_objects": 0, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "fast_read": false, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "options": {}, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "application_metadata": {}, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "read_balance": { 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "score_acting": 2.9900000095367432, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "score_stable": 2.9900000095367432, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_score": 0.67000001668930054, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_acting": 2, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "raw_score_stable": 2, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity_weighted": 1, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity": 1, 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: "average_primary_affinity_weighted": 1 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.552 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "osds": [ 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "d8cfa855-aa11-4830-a5ad-a736b4fc4569", 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 5, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 28, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 35, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 39, 2024-02-11T23:05:23.553 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 29, 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6802", 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6803", 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.554 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6804", 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6805", 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.555 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6808", 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6809", 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.556 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6806", 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6807", 2024-02-11T23:05:23.557 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2610045605 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6803/2610045605", 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6805/2610045605", 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2610045605", 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2610045605", 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.558 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "e418b766-1bc3-4cd6-b910-0d1070b7b9e2", 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 0, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 0, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 0, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 10, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 23, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 30, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.559 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.560 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.561 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:05:23.562 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 2724094408 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.563 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/2724094408", 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/2724094408", 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2724094408", 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2724094408", 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "exists" 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "uuid": "f360f5a2-846b-41c8-a82e-66347b28d104", 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 1, 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "in": 1, 2024-02-11T23:05:23.564 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 1, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_affinity": 1, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_begin": 15, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "last_clean_end": 28, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 37, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "up_thru": 0, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "down_at": 31, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "lost_at": 0, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addrs": { 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6810", 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.565 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6811", 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addrs": { 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6812", 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.566 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6813", 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6816", 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.567 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6817", 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "addrvec": [ 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v2", 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6814", 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.568 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "v1", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "addr": "127.0.0.1:6815", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "nonce": 3897082287 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "public_addr": "127.0.0.1:6811/3897082287", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "cluster_addr": "127.0.0.1:6813/3897082287", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3897082287", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3897082287", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "state": [ 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "exists", 2024-02-11T23:05:23.569 INFO:tasks.workunit.client.0.smithi121.stdout: "up" 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_xinfo": [ 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:55.355411+0000", 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:05:23.570 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:01:50.296898+0000", 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:55.944413+0000", 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:02:26.626900+0000", 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:05:23.571 INFO:tasks.workunit.client.0.smithi121.stdout: "down_stamp": "2024-02-11T23:03:56.525347+0000", 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_probability": 0, 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "laggy_interval": 0, 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "features": 4540701272860131327, 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "old_weight": 0, 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-11T23:03:03.950310+0000", 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "dead_epoch": 0 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap": [], 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_items": [], 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_upmap_primaries": [], 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "pg_temp": [], 2024-02-11T23:05:23.572 INFO:tasks.workunit.client.0.smithi121.stdout: "primary_temp": [], 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "blocklist": {}, 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "range_blocklist": {}, 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "erasure_code_profiles": { 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "default": { 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "k": "2", 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "m": "2", 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "plugin": "jerasure", 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "technique": "reed_sol_van" 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "removed_snaps_queue": [], 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "new_removed_snaps": [], 2024-02-11T23:05:23.573 INFO:tasks.workunit.client.0.smithi121.stdout: "new_purged_snaps": [], 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "crush_node_flags": {}, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "device_class_flags": {}, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode": { 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_metadata": [ 2024-02-11T23:05:23.574 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6804/2610045605,v1:127.0.0.1:6805/2610045605]", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:05:23.575 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:05:23.576 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:01:22.387769Z", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6802/2610045605,v1:127.0.0.1:6803/2610045605]", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2610045605,v1:127.0.0.1:6809/2610045605]", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2610045605,v1:127.0.0.1:6807/2610045605]", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:05:23.577 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/0", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:05:23.578 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/2724094408,v1:127.0.0.1:6813/2724094408]", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-02-11T23:05:23.579 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:01:58.183936Z", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:05:23.580 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/2724094408,v1:127.0.0.1:6811/2724094408]", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2724094408,v1:127.0.0.1:6817/2724094408]", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2724094408,v1:127.0.0.1:6815/2724094408]", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:05:23.581 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/1", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "arch": "x86_64", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "back_addr": "[v2:127.0.0.1:6812/3897082287,v1:127.0.0.1:6813/3897082287]", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "back_iface": "", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs": "1", 2024-02-11T23:05:23.582 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_db": "0", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_dedicated_wal": "0", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluefs_single_shared_device": "1", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_access_mode": "file", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_block_size": "4096", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_devices": "sda", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_rotational": "1", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_size": "107374182400", 2024-02-11T23:05:23.583 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_support_discard": "0", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_bdev_type": "hdd", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "bluestore_min_alloc_size": "4096", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_release": "squid", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_short": "19.0.0-658-gd4ed20cf", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "ceph_version_when_created": "ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev)", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "created_at": "2024-02-11T23:02:34.824016Z", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "default_device_class": "hdd", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QGN9", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": "sda", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "distro": "ubuntu", 2024-02-11T23:05:23.584 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "distro_version": "22.04", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "front_addr": "[v2:127.0.0.1:6810/3897082287,v1:127.0.0.1:6811/3897082287]", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "front_iface": "", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3897082287,v1:127.0.0.1:6817/3897082287]", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3897082287,v1:127.0.0.1:6815/3897082287]", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "hostname": "smithi121", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "journal_rotational": "1", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_description": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "kernel_version": "5.15.0-94-generic", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_swap_kb": "2097148", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "mem_total_kb": "32746608", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "objectstore_numa_unknown_devices": "sda", 2024-02-11T23:05:23.585 INFO:tasks.workunit.client.0.smithi121.stdout: "os": "Linux", 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_data": "td/divergent-priors/2", 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "osdspec_affinity": "", 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "rotational": "1" 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_clean_epochs": { 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "min_last_epoch_clean": 0, 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "last_epoch_clean": { 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "per_pool": [] 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_epochs": [ 2024-02-11T23:05:23.586 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 35 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 10 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 37 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.587 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_first_committed": 1, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "osdmap_last_committed": 40, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "crushmap": { 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "devices": [ 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.0", 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.1", 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.588 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd.2", 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "class": "hdd" 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "types": [ 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 0, 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "osd" 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "host" 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.589 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 2, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "chassis" 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 3, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "rack" 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 4, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "row" 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 5, 2024-02-11T23:05:23.590 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pdu" 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 6, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "pod" 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 7, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "room" 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 8, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "datacenter" 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.591 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 9, 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "zone" 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 10, 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "region" 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "root" 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: "buckets": [ 2024-02-11T23:05:23.592 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -1, 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default", 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.593 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -2, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "default~hdd", 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 11, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "root", 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.594 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -3, 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121", 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:05:23.595 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.596 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "id": -4, 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "name": "smithi121~hdd", 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "type_id": 1, 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "type_name": "host", 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 19206, 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "alg": "straw2", 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "hash": "rjenkins1", 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "items": [ 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.597 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 0, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 0 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 1, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 1 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "id": 2, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "weight": 6402, 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: "pos": 2 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.598 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "rules": [ 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_id": 0, 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "rule_name": "replicated_rule", 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "type": 1, 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "steps": [ 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "take", 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "item": -1, 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "item_name": "default" 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "choose_firstn", 2024-02-11T23:05:23.599 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 0, 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "type": "osd" 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "op": "emit" 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: ] 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "tunables": { 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_tries": 0, 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_local_fallback_tries": 0, 2024-02-11T23:05:23.600 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_total_tries": 50, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_descend_once": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_vary_r": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "chooseleaf_stable": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "straw_calc_version": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "allowed_bucket_algs": 54, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "profile": "jewel", 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "optimal_tunables": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "legacy_tunables": 0, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "minimum_required_version": "jewel", 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables2": 1, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v2_rules": 0, 2024-02-11T23:05:23.601 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables3": 1, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v3_rules": 0, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v4_buckets": 1, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "require_feature_tunables5": 1, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "has_v5_rules": 0 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "choose_args": {} 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "fsmap": { 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 1, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "default_fscid": -1, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": { 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "compat": {}, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "ro_compat": {}, 2024-02-11T23:05:23.602 INFO:tasks.workunit.client.0.smithi121.stdout: "incompat": { 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_1": "base v0.20", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_2": "client writeable ranges", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_3": "default file layouts on dirs", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_4": "dir inode in separate object", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_5": "mds uses versioned encoding", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_6": "dirfrag is stored in omap", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_8": "no anchor table", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_9": "file layout v2", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_10": "snaprealm v2", 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_11": "minor log segments" 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.603 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "feature_flags": { 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "enable_multiple": true, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "ever_enabled_multiple": true 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "standbys": [], 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "filesystems": [] 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_first_committed": 1, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "mdsmap_last_committed": 1, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "auth": { 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 5, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "num_secrets": 3 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.604 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_active": 1, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_unknown": 0, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osd": 3, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_sum": { 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 62050, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:05:23.605 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 101, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 101, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:05:23.606 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:05:23.607 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:05:23.608 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 17, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 17, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 0 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum": { 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:05:23.609 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83596, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2080, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489204, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036944896, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2129920, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 450986, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:05:23.610 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10397, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416931 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:05:23.611 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_sum_by_class": { 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "hdd": { 2024-02-11T23:05:23.612 INFO:tasks.workunit.client.0.smithi121.stdout: "up_from": 0, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 0, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pgs": 3, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "num_osds": 3, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_osds": 3, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "num_per_pool_omap_osds": 3, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "kb": 314572800, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used": 83596, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_data": 2080, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_omap": 10, 2024-02-11T23:05:23.613 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_used_meta": 81461, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "kb_avail": 314489204, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "statfs": { 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 322122547200, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 322036944896, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 2129920, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 450986, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 10397, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 83416931 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.614 INFO:tasks.workunit.client.0.smithi121.stdout: "hb_peers": [], 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "snap_trim_queue_len": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "num_snap_trimming": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shards_repaired": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "op_queue_age_hist": { 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "histogram": [], 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "upper_bound": 1 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "perf_stat": { 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ms": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ms": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "commit_latency_ns": 0, 2024-02-11T23:05:23.615 INFO:tasks.workunit.client.0.smithi121.stdout: "apply_latency_ns": 0 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "alerts": [], 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "network_ping_times": [] 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "pool_stats": [ 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "poolid": 1, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg": 1, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "stat_sum": { 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes": 62050, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects": 100, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_clones": 0, 2024-02-11T23:05:23.616 INFO:tasks.workunit.client.0.smithi121.stdout: "num_object_copies": 300, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing_on_primary": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_missing": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_degraded": 100, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_misplaced": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_unfound": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_dirty": 100, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_whiteouts": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_read_kb": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write": 101, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_write_kb": 101, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_scrub_errors": 0, 2024-02-11T23:05:23.617 INFO:tasks.workunit.client.0.smithi121.stdout: "num_shallow_scrub_errors": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_deep_scrub_errors": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_recovered": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_recovered": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_keys_recovered": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_omap": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_hit_set_archive": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_bytes_hit_set_archive": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_kb": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_kb": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_promote": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_high": 0, 2024-02-11T23:05:23.618 INFO:tasks.workunit.client.0.smithi121.stdout: "num_flush_mode_low": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_some": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_evict_mode_full": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_pinned": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_legacy_snapsets": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_large_omap_objects": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_manifest": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_bytes": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_omap_keys": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "num_objects_repaired": 0 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "store_stats": { 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "total": 0, 2024-02-11T23:05:23.619 INFO:tasks.workunit.client.0.smithi121.stdout: "available": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "internally_reserved": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "allocated": 1220608, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "data_stored": 185266, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_allocated": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "data_compressed_original": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "omap_allocated": 0, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "internal_metadata": 0 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "log_size": 17, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "ondisk_log_size": 17, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "up": 2, 2024-02-11T23:05:23.620 INFO:tasks.workunit.client.0.smithi121.stdout: "acting": 2, 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "num_store_stats": 3 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "osd_stats": [ 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 150323855380 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 42949672990 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.621 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "seq": 158913789971 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_state": [ 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "state": "active+undersized+degraded", 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "num": 1 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "num_pg_by_osd": [ 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 2, 2024-02-11T23:05:23.622 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 0, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 1, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 1, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "osd": 1, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_primary_pg": 0, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_acting_pg": 0, 2024-02-11T23:05:23.623 INFO:tasks.workunit.client.0.smithi121.stdout: "num_up_not_acting_pg": 0 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": [ 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: { 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "pool": 1, 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "purged_snaps": {} 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: ], 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "servicemap": { 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "epoch": 7, 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "modified": "2024-02-11T23:05:08.779318+0000", 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: "services": {} 2024-02-11T23:05:23.624 INFO:tasks.workunit.client.0.smithi121.stdout: }, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_first_committed": 1, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "mgrstat_last_committed": 113, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_first_committed": 1, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "logm_last_committed": 446, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "paxos": { 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "first_committed": 1, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "last_committed": 650, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "last_pn": 0, 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: "accepted_pn": 0 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout: } 2024-02-11T23:05:23.625 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:05:23.626 INFO:tasks.workunit.client.0.smithi121.stderr:report 1687823668 2024-02-11T23:05:23.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-02-11T23:05:23.626 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-02-11T23:05:23.626 INFO:tasks.workunit.client.0.smithi121.stdout:delay recovery 2024-02-11T23:05:23.627 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:23.944 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:23.945 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:122 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:23.945 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:23.945 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:23.945 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:23.960 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:05:23.960 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-02-11T23:05:23.960 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-11T23:05:23.960 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-11T23:05:23.961 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:05:23.961 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:05:23.961 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:05:23.961 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.0.asok 2024-02-11T23:05:23.962 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:05:23.962 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.0.asok set_recovery_delay 100000 2024-02-11T23:05:24.059 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:05:24.069 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-02-11T23:05:24.071 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-02-11T23:05:24.071 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-02-11T23:05:24.071 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-02-11T23:05:24.071 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:05:24.071 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:05:24.072 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:05:24.072 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.2.asok 2024-02-11T23:05:24.072 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:05:24.073 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.2.asok set_recovery_delay 100000 2024-02-11T23:05:24.178 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:05:24.189 INFO:tasks.workunit.client.0.smithi121.stdout:revive divergent 1 2024-02-11T23:05:24.189 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-02-11T23:05:24.190 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:24.564 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:24.564 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:122 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:24.564 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:24.564 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:24.564 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:24.579 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-02-11T23:05:25.176 INFO:tasks.workunit.client.0.smithi121.stderr:noup is set 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:05:25.195 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:05:25.196 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:05:25.198 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-11T23:05:25.199 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:05:25.200 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:05:25.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:25.237+0000 7f01f3802640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:25.251 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:25.249+0000 7f01f3802640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:25.252 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:25.249+0000 7f01f3802640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:25.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-02-11T23:05:25.674 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-02-11T23:05:25.792 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:25.789+0000 7f01f3802640 -1 Falling back to public interface 2024-02-11T23:05:28.845 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:28.841+0000 7f01f3802640 -1 osd.1 28 log_to_monitors true 2024-02-11T23:05:30.676 INFO:tasks.workunit.client.0.smithi121.stdout:delay recovery divergent 2024-02-11T23:05:30.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-02-11T23:05:30.676 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:31.062 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:31.062 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:123 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:31.062 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:31.062 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:31.062 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:31.077 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-02-11T23:05:31.077 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-11T23:05:31.077 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-11T23:05:31.078 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:05:31.078 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:05:31.078 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:05:31.079 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.1.asok 2024-02-11T23:05:31.079 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:05:31.079 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.1.asok set_recovery_delay 100000 2024-02-11T23:05:31.192 INFO:tasks.workunit.client.0.smithi121.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-02-11T23:05:31.206 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-02-11T23:05:31.965 INFO:tasks.workunit.client.0.smithi121.stderr:noup is unset 2024-02-11T23:05:31.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-02-11T23:05:31.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:05:31.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-11T23:05:31.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:05:31.984 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:05:31.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:05:31.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:05:31.985 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:05:31.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:05:31.985 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-11T23:05:32.461 INFO:tasks.workunit.client.0.smithi121.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2610045605,v1:127.0.0.1:6803/2610045605] [v2:127.0.0.1:6804/2610045605,v1:127.0.0.1:6805/2610045605] exists,up d8cfa855-aa11-4830-a5ad-a736b4fc4569 2024-02-11T23:05:32.462 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:05:32.462 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:05:32.463 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:05:32.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:05:32.464 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:05:32.926 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/2247714174,v1:127.0.0.1:6819/2247714174] [v2:127.0.0.1:6820/2247714174,v1:127.0.0.1:6821/2247714174] exists,up e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:05:32.928 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:05:32.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:05:32.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:05:32.929 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:05:32.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:05:32.929 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-11T23:05:33.330 INFO:tasks.workunit.client.0.smithi121.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3897082287,v1:127.0.0.1:6811/3897082287] [v2:127.0.0.1:6812/3897082287,v1:127.0.0.1:6813/3897082287] exists,up f360f5a2-846b-41c8-a82e-66347b28d104 2024-02-11T23:05:33.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:05:33.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:05:33.330 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:05:33.331 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:33.716 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:33.716 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:123 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:33.716 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:33.716 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:33.716 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:33.731 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-02-11T23:05:33.731 INFO:tasks.workunit.client.0.smithi121.stdout:wait for peering 2024-02-11T23:05:33.732 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:34.095 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:34.095 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:123 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:34.095 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:34.095 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:34.095 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:34.111 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-02-11T23:05:34.251 INFO:tasks.workunit.client.0.smithi121.stdout:killing divergent 1 2024-02-11T23:05:34.251 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-02-11T23:05:34.252 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:34.639 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:34.640 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-02-11T23:04:10.385969+0000 40'101 40:123 [0,2] 0 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 periodic scrub scheduled @ 2024-02-13T05:22:17.955517+0000 0 0 2024-02-11T23:05:34.640 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:34.640 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:34.640 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:34.655 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-02-11T23:05:34.655 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:05:34.655 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:05:34.656 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:05:34.656 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:05:34.656 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:05:34.762 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:05:34.762 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.26067.out 2024-02-11T23:05:34.762 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26067.out 2024-02-11T23:05:34.762 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-02-11T23:05:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-11T23:05:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-11T23:05:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-11T23:05:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-02-11T23:05:34.763 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26067.out 2024-02-11T23:05:35.930 INFO:tasks.workunit.client.0.smithi121.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-02-11T23:05:35.932 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:023dd3d2:::existing_12:head# 2024-02-11T23:05:35.933 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:04c40a0c:::existing_93:head# 2024-02-11T23:05:35.934 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:064ff5fb:::existing_36:head# 2024-02-11T23:05:35.934 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:091f9f1d:::existing_94:head# 2024-02-11T23:05:35.934 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:0a865a93:::existing_54:head# 2024-02-11T23:05:35.935 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:0fed513c:::existing_41:head# 2024-02-11T23:05:35.935 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-02-11T23:05:35.935 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:169157fa:::existing_81:head# 2024-02-11T23:05:35.936 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:17731382:::existing_58:head# 2024-02-11T23:05:35.937 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:18a45a39:::existing_43:head# 2024-02-11T23:05:35.937 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:1c160f01:::existing_66:head# 2024-02-11T23:05:35.938 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:1dd6c249:::existing_97:head# 2024-02-11T23:05:35.938 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:227e0c77:::existing_7:head# 2024-02-11T23:05:35.939 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:23e1b063:::existing_59:head# 2024-02-11T23:05:35.941 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:28566edb:::existing_78:head# 2024-02-11T23:05:35.941 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:2a101182:::existing_20:head# 2024-02-11T23:05:35.941 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:2d1519da:::existing_96:head# 2024-02-11T23:05:35.941 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:3078b87a:::existing_86:head# 2024-02-11T23:05:35.941 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:30bde01b:::existing_15:head# 2024-02-11T23:05:35.942 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:317a7e19:::existing_56:head# 2024-02-11T23:05:35.942 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-02-11T23:05:35.943 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:3e16d84b:::existing_48:head# 2024-02-11T23:05:35.943 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:45fe8203:::existing_75:head# 2024-02-11T23:05:35.944 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-02-11T23:05:35.944 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:5298dcdb:::existing_57:head# 2024-02-11T23:05:35.945 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:52ebaa44:::existing_32:head# 2024-02-11T23:05:35.945 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:53588912:::existing_9:head# 2024-02-11T23:05:35.946 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-02-11T23:05:35.947 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:5da0255c:::existing_13:head# 2024-02-11T23:05:35.947 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:5da2dab8:::existing_53:head# 2024-02-11T23:05:35.947 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:5ee3514b:::existing_30:head# 2024-02-11T23:05:35.947 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:602f83fe:::foo:head# 2024-02-11T23:05:35.948 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:60426cc7:::existing_62:head# 2024-02-11T23:05:35.948 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:628918ca:::existing_79:head# 2024-02-11T23:05:35.949 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-02-11T23:05:35.949 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:67e1780f:::existing_27:head# 2024-02-11T23:05:35.950 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:68143ca1:::existing_71:head# 2024-02-11T23:05:35.950 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:6bac9509:::existing_29:head# 2024-02-11T23:05:35.951 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:6bbec798:::existing_45:head# 2024-02-11T23:05:35.952 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:6cb23a40:::existing_65:head# 2024-02-11T23:05:35.952 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:7055ec5e:::existing_83:head# 2024-02-11T23:05:35.953 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:70a54e2b:::existing_87:head# 2024-02-11T23:05:35.954 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:750ace21:::existing_19:head# 2024-02-11T23:05:35.954 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:78623b1d:::existing_6:head# 2024-02-11T23:05:35.955 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:79881152:::existing_63:head# 2024-02-11T23:05:35.955 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-02-11T23:05:35.956 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:81490677:::existing_76:head# 2024-02-11T23:05:35.957 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-02-11T23:05:35.957 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-02-11T23:05:35.958 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-02-11T23:05:35.958 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:923c5d4d:::existing_5:head# 2024-02-11T23:05:35.959 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:92731e71:::existing_89:head# 2024-02-11T23:05:35.959 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:928205cf:::existing_82:head# 2024-02-11T23:05:35.960 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:93be2267:::existing_24:head# 2024-02-11T23:05:35.961 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:99ceda2c:::existing_44:head# 2024-02-11T23:05:35.961 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:9dd917ee:::existing_26:head# 2024-02-11T23:05:35.962 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a28805a6:::existing_72:head# 2024-02-11T23:05:35.963 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a294a8f6:::existing_16:head# 2024-02-11T23:05:35.963 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a2f5473a:::existing_42:head# 2024-02-11T23:05:35.964 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a504a533:::existing_90:head# 2024-02-11T23:05:35.964 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a691337e:::existing_61:head# 2024-02-11T23:05:35.965 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-02-11T23:05:35.965 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:a9fde86e:::existing_18:head# 2024-02-11T23:05:35.966 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:aa309faa:::existing_68:head# 2024-02-11T23:05:35.967 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-02-11T23:05:35.967 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:b279b1fe:::existing_10:head# 2024-02-11T23:05:35.968 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:b343e277:::existing_88:head# 2024-02-11T23:05:35.968 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:b7130e90:::existing_80:head# 2024-02-11T23:05:35.969 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:bc6f7309:::existing_70:head# 2024-02-11T23:05:35.970 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:be8489b7:::existing_38:head# 2024-02-11T23:05:35.970 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-02-11T23:05:35.971 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:bebec877:::existing_74:head# 2024-02-11T23:05:35.971 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:c04347d8:::existing_60:head# 2024-02-11T23:05:35.972 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:c156033e:::existing_100:head# 2024-02-11T23:05:35.972 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-02-11T23:05:35.973 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-02-11T23:05:35.974 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-02-11T23:05:35.974 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-02-11T23:05:35.975 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:ccb3865b:::existing_73:head# 2024-02-11T23:05:35.975 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:cd84e19e:::existing_40:head# 2024-02-11T23:05:35.976 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-02-11T23:05:35.977 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:d41a25c9:::existing_22:head# 2024-02-11T23:05:35.977 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:d7f088da:::existing_91:head# 2024-02-11T23:05:35.978 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-02-11T23:05:35.979 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-02-11T23:05:35.979 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:dd677bd3:::existing_35:head# 2024-02-11T23:05:35.980 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-02-11T23:05:35.980 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:e0e22077:::existing_21:head# 2024-02-11T23:05:35.981 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:e255b710:::existing_55:head# 2024-02-11T23:05:35.982 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:eff1d5db:::existing_14:head# 2024-02-11T23:05:35.982 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:f1b156bf:::existing_23:head# 2024-02-11T23:05:35.983 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:f5ce4199:::existing_34:head# 2024-02-11T23:05:35.983 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:f6809456:::existing_11:head# 2024-02-11T23:05:35.984 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:f699a2a9:::existing_67:head# 2024-02-11T23:05:35.984 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:f9baac43:::existing_25:head# 2024-02-11T23:05:35.985 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:fb76688e:::existing_51:head# 2024-02-11T23:05:35.986 INFO:tasks.workunit.client.0.smithi121.stderr:Read #1:fbb92eec:::existing_69:head# 2024-02-11T23:05:35.986 INFO:tasks.workunit.client.0.smithi121.stderr:Export successful 2024-02-11T23:05:35.987 INFO:tasks.workunit.client.0.smithi121.stdout: marking collection for removal 2024-02-11T23:05:35.987 INFO:tasks.workunit.client.0.smithi121.stdout:setting '_remove' omap key 2024-02-11T23:05:35.987 INFO:tasks.workunit.client.0.smithi121.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-02-11T23:05:36.246 INFO:tasks.workunit.client.0.smithi121.stderr:Remove successful 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26067.out 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-02-11T23:05:36.784 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26067.out 2024-02-11T23:05:37.786 INFO:tasks.workunit.client.0.smithi121.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-02-11T23:05:37.787 INFO:tasks.workunit.client.0.smithi121.stdout:Importing pgid 1.0 2024-02-11T23:05:37.787 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:023dd3d2:::existing_12:head# 2024-02-11T23:05:37.787 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:37.891 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:04c40a0c:::existing_93:head# 2024-02-11T23:05:37.891 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:37.941 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:064ff5fb:::existing_36:head# 2024-02-11T23:05:37.942 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:37.975 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:091f9f1d:::existing_94:head# 2024-02-11T23:05:37.975 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.008 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:0a865a93:::existing_54:head# 2024-02-11T23:05:38.009 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.042 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:0fed513c:::existing_41:head# 2024-02-11T23:05:38.042 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.075 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-02-11T23:05:38.075 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.109 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:169157fa:::existing_81:head# 2024-02-11T23:05:38.109 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.142 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:17731382:::existing_58:head# 2024-02-11T23:05:38.142 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.176 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:18a45a39:::existing_43:head# 2024-02-11T23:05:38.176 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.209 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:1c160f01:::existing_66:head# 2024-02-11T23:05:38.209 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.242 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:1dd6c249:::existing_97:head# 2024-02-11T23:05:38.242 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.276 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:227e0c77:::existing_7:head# 2024-02-11T23:05:38.276 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.309 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:23e1b063:::existing_59:head# 2024-02-11T23:05:38.309 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.343 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:28566edb:::existing_78:head# 2024-02-11T23:05:38.343 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.376 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:2a101182:::existing_20:head# 2024-02-11T23:05:38.376 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.410 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:2d1519da:::existing_96:head# 2024-02-11T23:05:38.410 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.443 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:3078b87a:::existing_86:head# 2024-02-11T23:05:38.443 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.485 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:30bde01b:::existing_15:head# 2024-02-11T23:05:38.485 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.519 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:317a7e19:::existing_56:head# 2024-02-11T23:05:38.519 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.552 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-02-11T23:05:38.552 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.586 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:3e16d84b:::existing_48:head# 2024-02-11T23:05:38.586 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.619 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:45fe8203:::existing_75:head# 2024-02-11T23:05:38.619 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.653 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-02-11T23:05:38.653 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.686 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:5298dcdb:::existing_57:head# 2024-02-11T23:05:38.686 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.720 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:52ebaa44:::existing_32:head# 2024-02-11T23:05:38.720 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.753 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:53588912:::existing_9:head# 2024-02-11T23:05:38.753 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.786 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-02-11T23:05:38.787 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.820 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:5da0255c:::existing_13:head# 2024-02-11T23:05:38.820 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.863 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:5da2dab8:::existing_53:head# 2024-02-11T23:05:38.863 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.912 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:5ee3514b:::existing_30:head# 2024-02-11T23:05:38.912 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:38.962 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:602f83fe:::foo:head# 2024-02-11T23:05:38.962 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.004 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:60426cc7:::existing_62:head# 2024-02-11T23:05:39.004 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.046 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:628918ca:::existing_79:head# 2024-02-11T23:05:39.046 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.087 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-02-11T23:05:39.088 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.129 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:67e1780f:::existing_27:head# 2024-02-11T23:05:39.130 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.171 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:68143ca1:::existing_71:head# 2024-02-11T23:05:39.171 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.213 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:6bac9509:::existing_29:head# 2024-02-11T23:05:39.213 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.255 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:6bbec798:::existing_45:head# 2024-02-11T23:05:39.255 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.296 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:6cb23a40:::existing_65:head# 2024-02-11T23:05:39.297 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.338 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:7055ec5e:::existing_83:head# 2024-02-11T23:05:39.338 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.380 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:70a54e2b:::existing_87:head# 2024-02-11T23:05:39.380 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.422 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:750ace21:::existing_19:head# 2024-02-11T23:05:39.422 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.489 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:78623b1d:::existing_6:head# 2024-02-11T23:05:39.489 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.564 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:79881152:::existing_63:head# 2024-02-11T23:05:39.564 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.623 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-02-11T23:05:39.623 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.665 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:81490677:::existing_76:head# 2024-02-11T23:05:39.665 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.706 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-02-11T23:05:39.707 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.748 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-02-11T23:05:39.748 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.790 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-02-11T23:05:39.790 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.832 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:923c5d4d:::existing_5:head# 2024-02-11T23:05:39.832 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.874 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:92731e71:::existing_89:head# 2024-02-11T23:05:39.874 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.915 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:928205cf:::existing_82:head# 2024-02-11T23:05:39.916 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:39.974 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:93be2267:::existing_24:head# 2024-02-11T23:05:39.974 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.016 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:99ceda2c:::existing_44:head# 2024-02-11T23:05:40.016 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.049 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:9dd917ee:::existing_26:head# 2024-02-11T23:05:40.049 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.082 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a28805a6:::existing_72:head# 2024-02-11T23:05:40.083 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.116 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a294a8f6:::existing_16:head# 2024-02-11T23:05:40.116 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.149 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a2f5473a:::existing_42:head# 2024-02-11T23:05:40.150 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.183 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a504a533:::existing_90:head# 2024-02-11T23:05:40.183 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.216 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a691337e:::existing_61:head# 2024-02-11T23:05:40.216 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.250 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-02-11T23:05:40.250 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.283 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:a9fde86e:::existing_18:head# 2024-02-11T23:05:40.283 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.317 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:aa309faa:::existing_68:head# 2024-02-11T23:05:40.317 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.351 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-02-11T23:05:40.351 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.409 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:b279b1fe:::existing_10:head# 2024-02-11T23:05:40.409 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.442 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:b343e277:::existing_88:head# 2024-02-11T23:05:40.442 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.476 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:b7130e90:::existing_80:head# 2024-02-11T23:05:40.476 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.509 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:bc6f7309:::existing_70:head# 2024-02-11T23:05:40.509 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.543 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:be8489b7:::existing_38:head# 2024-02-11T23:05:40.543 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.576 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-02-11T23:05:40.576 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.611 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:bebec877:::existing_74:head# 2024-02-11T23:05:40.611 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.643 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:c04347d8:::existing_60:head# 2024-02-11T23:05:40.643 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.676 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:c156033e:::existing_100:head# 2024-02-11T23:05:40.676 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.710 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-02-11T23:05:40.710 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.744 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-02-11T23:05:40.744 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.777 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-02-11T23:05:40.777 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.810 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-02-11T23:05:40.811 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.844 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:ccb3865b:::existing_73:head# 2024-02-11T23:05:40.844 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.903 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:cd84e19e:::existing_40:head# 2024-02-11T23:05:40.903 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.953 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-02-11T23:05:40.953 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:40.986 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:d41a25c9:::existing_22:head# 2024-02-11T23:05:40.986 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.020 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:d7f088da:::existing_91:head# 2024-02-11T23:05:41.020 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.053 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-02-11T23:05:41.053 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.087 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-02-11T23:05:41.087 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.120 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:dd677bd3:::existing_35:head# 2024-02-11T23:05:41.120 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.154 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-02-11T23:05:41.154 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.187 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:e0e22077:::existing_21:head# 2024-02-11T23:05:41.187 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.223 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:e255b710:::existing_55:head# 2024-02-11T23:05:41.223 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.254 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:eff1d5db:::existing_14:head# 2024-02-11T23:05:41.254 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.287 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:f1b156bf:::existing_23:head# 2024-02-11T23:05:41.287 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.321 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:f5ce4199:::existing_34:head# 2024-02-11T23:05:41.321 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.354 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:f6809456:::existing_11:head# 2024-02-11T23:05:41.354 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.388 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:f699a2a9:::existing_67:head# 2024-02-11T23:05:41.388 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.421 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:f9baac43:::existing_25:head# 2024-02-11T23:05:41.421 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.488 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:fb76688e:::existing_51:head# 2024-02-11T23:05:41.488 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.547 INFO:tasks.workunit.client.0.smithi121.stdout:Write #1:fbb92eec:::existing_69:head# 2024-02-11T23:05:41.547 INFO:tasks.workunit.client.0.smithi121.stdout:snapset 0=[]:{} 2024-02-11T23:05:41.597 INFO:tasks.workunit.client.0.smithi121.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-02-11T23:05:41.597 INFO:tasks.workunit.client.0.smithi121.stderr:*** Caught signal (Segmentation fault) ** 2024-02-11T23:05:41.597 INFO:tasks.workunit.client.0.smithi121.stderr: in thread 7fd8c7756580 thread_name:ceph-objectstor 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: ceph version 19.0.0-658-gd4ed20cf (d4ed20cfb238ef2c47d64ca4533eca00b8968b98) squid (dev) 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd8c8518520] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 2: (PGLog::_write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, eversion_t, eversion_t, eversion_t, std::set, std::allocator >&&, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >&&, pg_missing_set const&, bool, bool, bool, eversion_t, eversion_t, eversion_t, bool*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >*, DoutPrefixProvider const*)+0x867) [0x558f532d9707] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 3: (PGLog::write_log_and_missing(ceph::common::CephContext*, ceph::os::Transaction&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t const&, pg_missing_set const&, bool, bool*, DoutPrefixProvider const*)+0x178) [0x558f532c1828] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 4: (write_pg(ceph::os::Transaction&, unsigned int, pg_info_t&, pg_log_t&, PastIntervals&, std::map, std::allocator > >&, pg_missing_set&)+0x4f9) [0x558f53173bc9] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::__cxx11::basic_string, std::allocator >)+0x24ce) [0x558f5318bbfe] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 6: main() 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 7: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fd8c84ffd90] 2024-02-11T23:05:41.600 INFO:tasks.workunit.client.0.smithi121.stderr: 8: __libc_start_main() 2024-02-11T23:05:41.601 INFO:tasks.workunit.client.0.smithi121.stderr: 9: _start() 2024-02-11T23:05:41.759 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1256: 51644 Segmentation fault (core dumped) ceph-objectstore-tool --data-path $osd_data "$@" 2024-02-11T23:05:41.759 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-02-11T23:05:41.759 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-02-11T23:05:41.760 INFO:tasks.workunit.client.0.smithi121.stdout:reviving divergent 1 2024-02-11T23:05:41.760 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:05:42.159 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:05:42.159 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-02-11T23:05:33.826350+0000 40'101 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 no scrub is scheduled 0 0 2024-02-11T23:05:42.159 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:05:42.159 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:05:42.160 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:05:42.175 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-11T23:05:42.176 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-11T23:05:42.177 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-02-11T23:05:42.177 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-11T23:05:42.177 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-11T23:05:42.177 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-02-11T23:05:42.178 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-02-11T23:05:42.179 INFO:tasks.workunit.client.0.smithi121.stdout:start osd.1 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c15e6ddc-69cf-4a83-aaf1-df560b419426 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26067/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-11T23:05:42.180 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-11T23:05:42.218 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:42.217+0000 7fd9e2648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:42.220 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:42.217+0000 7fd9e2648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:42.221 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:42.221+0000 7fd9e2648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-11T23:05:42.631 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-11T23:05:42.631 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:05:42.632 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:05:42.633 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:05:42.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:42.789+0000 7fd9e2648640 -1 Falling back to public interface 2024-02-11T23:05:43.049 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/2247714174,v1:127.0.0.1:6819/2247714174] [v2:127.0.0.1:6820/2247714174,v1:127.0.0.1:6821/2247714174] exists,up e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-11T23:05:43.050 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stdout:0 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-11T23:05:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-11T23:05:43.488 INFO:tasks.workunit.client.0.smithi121.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/2247714174,v1:127.0.0.1:6819/2247714174] [v2:127.0.0.1:6820/2247714174,v1:127.0.0.1:6821/2247714174] exists,up e418b766-1bc3-4cd6-b910-0d1070b7b9e2 2024-02-11T23:05:43.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-11T23:05:43.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-11T23:05:43.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-11T23:05:43.489 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-02-11T23:05:46.033 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-11T23:05:46.033+0000 7fd9e2648640 -1 osd.1 47 log_to_monitors true 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:06:03.492 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26067/ceph-osd.1.asok 2024-02-11T23:06:03.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-02-11T23:06:03.493 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26067/ceph-osd.1.asok dump_ops_in_flight 2024-02-11T23:06:03.611 INFO:tasks.workunit.client.0.smithi121.stdout:{ 2024-02-11T23:06:03.611 INFO:tasks.workunit.client.0.smithi121.stdout: "ops": [], 2024-02-11T23:06:03.611 INFO:tasks.workunit.client.0.smithi121.stdout: "num_ops": 0 2024-02-11T23:06:03.611 INFO:tasks.workunit.client.0.smithi121.stdout:} 2024-02-11T23:06:03.624 INFO:tasks.workunit.client.0.smithi121.stdout:allowing recovery 2024-02-11T23:06:03.625 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-02-11T23:06:03.625 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:06:04.026 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:06:04.026 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded+remapped+backfilling 2024-02-11T23:05:48.265978+0000 47'102 53:137 [1,0,2] 1 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 no scrub is scheduled 0 0 2024-02-11T23:06:04.026 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:06:04.026 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:06:04.026 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:06:04.042 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-02-11T23:06:04.042 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-02-11T23:06:04.195 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:06:04.209 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-02-11T23:06:04.210 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-02-11T23:06:04.368 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:06:04.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-02-11T23:06:04.383 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-02-11T23:06:04.623 INFO:tasks.workunit.client.0.smithi121.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-02-11T23:06:04.642 INFO:tasks.workunit.client.0.smithi121.stdout:reading divergent objects 2024-02-11T23:06:04.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-02-11T23:06:04.642 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-02-11T23:06:05.034 INFO:tasks.workunit.client.0.smithi121.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-11T23:06:05.035 INFO:tasks.workunit.client.0.smithi121.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded+remapped+backfilling 2024-02-11T23:05:48.265978+0000 47'102 53:137 [1,0,2] 1 [0,2] 0 0'0 2024-02-11T23:03:08.113604+0000 0'0 2024-02-11T23:03:08.113604+0000 0 0 no scrub is scheduled 0 0 2024-02-11T23:06:05.035 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-02-11T23:06:05.035 INFO:tasks.workunit.client.0.smithi121.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-11T23:06:05.035 INFO:tasks.workunit.client.0.smithi121.stderr:dumped pgs 2024-02-11T23:06:05.051 INFO:tasks.workunit.client.0.smithi121.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-02-11T23:06:05.052 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-02-11T23:06:05.053 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:06:05.053 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-02-11T23:06:05.108 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:06:05.108 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-02-11T23:06:05.149 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:06:05.149 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-02-11T23:06:05.186 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-02-11T23:06:05.186 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-02-11T23:06:05.233 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-02-11T23:06:05.234 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-02-11T23:06:05.234 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-02-11T23:06:05.266 INFO:tasks.workunit.client.0.smithi121.stderr:error getting test/newobject_1: (2) No such file or directory 2024-02-11T23:06:05.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-02-11T23:06:05.268 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-02-11T23:06:05.301 INFO:tasks.workunit.client.0.smithi121.stderr:error getting test/newobject_2: (2) No such file or directory 2024-02-11T23:06:05.302 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-02-11T23:06:05.304 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-02-11T23:06:05.305 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'103 (28'4) delete 1:70b3bf12:::existing_4:head by client.4633.0:1 2024-02-11T23:03:44.552368+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'103 (28'4) delete 1:70b3bf12:::existing_4:head by client.4633.0:1 2024-02-11T23:03:44.552368+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'103 last_divergent_update: 28'103 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-02-11T23:04:47.509593+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4625.0:1 2024-02-11T23:03:44.538264+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4625.0:1 2024-02-11T23:03:44.538264+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-02-11T23:06:05.309 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-02-11T23:03:44.560236+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-02-11T23:03:44.560236+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'106 last_divergent_update: 28'106 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-02-11T23:03:44.560236+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-02-11T23:03:44.542307+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-02-11T23:03:44.542307+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:06:05.310 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'102 last_divergent_update: 28'102 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-02-11T23:03:44.542307+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-02-11T23:03:44.559486+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-02-11T23:03:44.559486+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.311 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'105 last_divergent_update: 28'105 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-02-11T23:03:44.559486+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4632.0:1 2024-02-11T23:03:44.551379+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4632.0:1 2024-02-11T23:03:44.551379+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-02-11T23:06:05.312 INFO:tasks.workunit.client.0.smithi121.stdout:td/divergent-priors/osd.1.log:2024-02-11T23:05:33.745+0000 7f01d778c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-02-11T23:06:05.313 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-02-11T23:06:05.313 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-02-11T23:06:05.316 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-02-11T23:06:05.317 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-02-11T23:06:05.321 INFO:tasks.workunit.client.0.smithi121.stdout:success 2024-02-11T23:06:05.321 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-02-11T23:06:05.322 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.26067.out 2024-02-11T23:06:05.322 INFO:tasks.workunit.client.0.smithi121.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26067.out': No such file or directory 2024-02-11T23:06:05.322 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-02-11T23:06:05.322 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-02-11T23:06:05.323 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-02-11T23:06:05.905 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'test' does not exist 2024-02-11T23:06:05.920 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-02-11T23:06:05.921 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:06:05.921 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:06:05.921 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:06:05.921 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:06:05.922 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:06:06.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:06:06.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-02-11T23:06:06.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-02-11T23:06:06.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-11T23:06:06.033 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-02-11T23:06:06.034 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:06:06.034 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:06:06.034 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:06:06.034 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:06:06.034 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:06:06.038 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:06:06.038 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T23:06:06.039 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T23:06:06.039 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T23:06:06.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T23:06:06.040 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T23:06:06.040 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T23:06:06.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:06:06.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T23:06:06.041 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T23:06:06.041 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:06:06.042 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:06:06.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: cores=yes 2024-02-11T23:06:06.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:182: teardown: '[' -n '' ']' 2024-02-11T23:06:06.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' yes = yes -o '' = 1 ']' 2024-02-11T23:06:06.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-11T23:06:06.043 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-11T23:06:06.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/divergent-priors/mgr.x.log td/divergent-priors/mon.a.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log /home/ubuntu/cephtest/archive/log 2024-02-11T23:06:06.044 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-02-11T23:06:06.054 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T23:06:06.054 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:06:06.054 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:06:06.054 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26067 2024-02-11T23:06:06.055 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' yes = yes ']' 2024-02-11T23:06:06.055 INFO:tasks.workunit.client.0.smithi121.stdout:ERROR: Failure due to cores found 2024-02-11T23:06:06.056 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:201: teardown: echo 'ERROR: Failure due to cores found' 2024-02-11T23:06:06.056 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:202: teardown: '[' -n '' ']' 2024-02-11T23:06:06.056 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:205: teardown: return 1 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: return 1 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/divergent-priors 1 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-11T23:06:06.057 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-11T23:06:06.058 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-11T23:06:06.058 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-11T23:06:06.058 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-11T23:06:06.059 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-11T23:06:06.059 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-11T23:06:06.059 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-11T23:06:06.060 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-11T23:06:06.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:06:06.060 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-11T23:06:06.061 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-11T23:06:06.061 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T23:06:06.061 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: cores=yes 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:182: teardown: '[' -n '' ']' 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' yes = yes -o 1 = 1 ']' 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-11T23:06:06.063 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv 'td/divergent-priors/*.log' /home/ubuntu/cephtest/archive/log 2024-02-11T23:06:06.064 INFO:tasks.workunit.client.0.smithi121.stderr:mv: cannot stat 'td/divergent-priors/*.log': No such file or directory 2024-02-11T23:06:06.064 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-02-11T23:06:06.065 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-11T23:06:06.065 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-11T23:06:06.065 INFO:tasks.workunit.client.0.smithi121.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26067 2024-02-11T23:06:06.065 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26067 2024-02-11T23:06:06.066 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' yes = yes ']' 2024-02-11T23:06:06.066 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:201: teardown: echo 'ERROR: Failure due to cores found' 2024-02-11T23:06:06.066 INFO:tasks.workunit.client.0.smithi121.stdout:ERROR: Failure due to cores found 2024-02-11T23:06:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T23:06:06.067 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:202: teardown: '[' -n '' ']' 2024-02-11T23:06:06.067 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:205: teardown: return 1 2024-02-11T23:06:06.067 INFO:tasks.workunit.client.0.smithi121.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: return 1 2024-02-11T23:06:06.068 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-02-11T23:06:06.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-11T23:06:06.365 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/divergent-priors.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh' 2024-02-11T23:06:06.593 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4af782c8e85d484fa02a53dcfa8a885b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/divergent-priors.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh' 2024-02-11T23:06:06.596 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-11T23:06:06.605 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-11T23:06:06.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-11T23:06:06.657 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-11T23:06:06.667 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-11T23:06:06.667 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout:*hv01.front.sepi 44.4.53.2 2 u 53 64 377 0.056 -2.202 0.861 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout:+hv02.front.sepi 96.245.170.99 3 u 55 64 377 0.094 -1.310 0.691 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout:+hv03.front.sepi 74.6.168.72 3 u 50 64 377 0.076 +1.323 0.765 2024-02-11T23:06:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-02-11T23:06:06.689 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-11T23:06:06.698 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-11T23:06:06.699 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-11T23:06:06.709 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-11T23:06:06.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-11T23:06:06.791 INFO:teuthology.task.internal:Duration was 1262.227710 seconds 2024-02-11T23:06:06.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-11T23:06:06.802 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-11T23:06:06.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-11T23:06:06.839 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-11T23:06:06.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-02-11T23:06:06.840 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-11T23:06:06.889 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-11T23:06:06.890 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-11T23:06:06.971 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-11T23:06:06.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-11T23:06:07.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-11T23:06:07.078 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-11T23:06:07.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-11T23:06:07.089 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-11T23:06:07.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:06:07.145 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-02-11T23:06:07.171 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:06:07.217 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi121.front.sepia.ceph.com, flagging run as failed 2024-02-11T23:06:07.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-11T23:06:07.228 INFO:teuthology.task.internal:Transferring archived files... 2024-02-11T23:06:07.230 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954/remote/smithi121 2024-02-11T23:06:07.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-11T23:06:13.636 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-02-11T23:06:13.751 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954/remote/smithi121/coredump/1707692741.51644.core.gz: gzip compressed data, was "1707692741.51644.core", last modified: Sun Feb 11 23:06:08 2024, max compression, original size modulo 2^32 167505920 2024-02-11T23:06:13.751 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-02-11T23:06:14.424 INFO:teuthology.task.internal: core looks like: /tmp/tmpz3gpxpw4: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph-objectstore-tool', platform: 'x86_64' 2024-02-11T23:06:14.444 INFO:teuthology.task.internal: dump_program: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d 2024-02-11T23:06:14.445 DEBUG:teuthology.orchestra.run.smithi121:> which 'ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d' 2024-02-11T23:06:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T23:06:14.452 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 1: "which 'ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d'" 2024-02-11T23:06:14.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-11T23:06:14.472 INFO:teuthology.task.internal:Not uploading archives. 2024-02-11T23:06:14.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-11T23:06:14.488 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-11T23:06:14.489 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-11T23:06:14.498 INFO:teuthology.orchestra.run.smithi121.stdout: 658625 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 11 23:06 /home/ubuntu/cephtest 2024-02-11T23:06:14.498 INFO:teuthology.orchestra.run.smithi121.stdout: 658626 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Feb 11 23:06 /home/ubuntu/cephtest/archive 2024-02-11T23:06:14.498 INFO:teuthology.orchestra.run.smithi121.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 23:06 /home/ubuntu/cephtest/archive/syslog 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 658630 4 -rw-rw-rw- 1 ubuntu ubuntu 528 Feb 11 22:52 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 658629 172 -rw-rw-r-- 1 ubuntu ubuntu 173646 Feb 11 23:06 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 663750 156 -rw-rw-rw- 1 ubuntu ubuntu 159302 Feb 11 23:06 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 788834 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 11 23:06 /home/ubuntu/cephtest/archive/log 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 687548 20428 -rw-r--r-- 1 ubuntu ubuntu 20912897 Feb 11 23:06 /home/ubuntu/cephtest/archive/log/mgr.x.log 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 687055 60136 -rw-r--r-- 1 ubuntu ubuntu 61573966 Feb 11 23:06 /home/ubuntu/cephtest/archive/log/mon.a.log 2024-02-11T23:06:14.499 INFO:teuthology.orchestra.run.smithi121.stdout: 688642 3496 -rw-r--r-- 1 ubuntu ubuntu 3577934 Feb 11 23:06 /home/ubuntu/cephtest/archive/log/osd.2.log 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 688637 6696 -rw-r--r-- 1 ubuntu ubuntu 6856697 Feb 11 23:06 /home/ubuntu/cephtest/archive/log/osd.1.log 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 687771 5028 -rw-r--r-- 1 ubuntu ubuntu 5142074 Feb 11 23:06 /home/ubuntu/cephtest/archive/log/osd.0.log 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 658627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 23:05 /home/ubuntu/cephtest/archive/coredump 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 688643 17028 -rw------- 1 ubuntu ubuntu 167505920 Feb 11 23:05 /home/ubuntu/cephtest/archive/coredump/1707692741.51644.core 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 671347 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 11 22:53 /home/ubuntu/cephtest/mnt.0 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 673374 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 11 22:57 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 686656 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 11 22:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-11T23:06:14.500 INFO:teuthology.orchestra.run.smithi121.stdout: 686745 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 11 23:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-02-11T23:06:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T23:06:14.501 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-11T23:06:14.502 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 1: "which 'ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d'" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-11T23:06:14.510 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-11T23:06:14.523 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-11T23:06:14.539 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi121 with status 1: "which 'ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/d'" 2024-02-11T23:06:14.539 INFO:teuthology.nuke:Checking targets against current locks 2024-02-11T23:06:14.567 DEBUG:teuthology.nuke:shortname: smithi121 2024-02-11T23:06:14.568 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-11T23:06:14.587 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 22:36:38.515468', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhN9lKvvBNcZlCqGC20PdPNscr2lTLGWI4jiQomAfpzmgtfxw8ktVt0xlWefAIztg5Jn4xzzcfICbng9+KMtzE='} 2024-02-11T23:06:14.636 INFO:teuthology.orchestra.console.smithi121:Power off 2024-02-11T23:06:14.637 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-11T23:06:14.662 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-02-11T23:06:14.662 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T23:06:14.688 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-11T23:06:18.689 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T23:06:18.799 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-11T23:06:22.804 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T23:06:22.915 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-11T23:06:26.916 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T23:06:27.027 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-02-11T23:06:27.127 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-02-11T23:06:27.233 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 1262.227709531784 failure_reason: 'Command failed (workunit test osd/divergent-priors.sh) on smithi121 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh''' flavor: default owner: scheduled_nmordech@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4af782c8e85d484fa02a53dcfa8a885b status: fail success: false 2024-02-11T23:06:27.234 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-11T23:06:27.355 INFO:teuthology.run:FAIL