2023-03-20T18:35:26.883 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-20T18:35:28.193 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146 --description rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} --name yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/orig.config.yaml 2023-03-20T18:35:28.226 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-20T18:35:29.967 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146 branch: wip-yuri3-testing-2023-03-17-1235-quincy description: rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} email: null first_in_suite: false job_id: '7213146' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-03-17-1235-quincy ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: c5c2494ca442a56f1a277b51de1b8b1a422ded18 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c5c2494ca442a56f1a277b51de1b8b1a422ded18 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: a6dd66830c12498d02f3894e352dd8b49a7bab4c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: c5c2494ca442a56f1a277b51de1b8b1a422ded18 sleep_before_teardown: 0 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a6dd66830c12498d02f3894e352dd8b49a7bab4c targets: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 7f7f7c33a372789d0252bbbcd5798300b0f044d0 timestamp: 2023-03-18_00:57:11 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4737 2023-03-20T18:35:30.821 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa; will attempt to use it 2023-03-20T18:35:31.859 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks 2023-03-20T18:35:34.101 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-20T18:35:34.103 INFO:teuthology.task.internal:Checking packages... 2023-03-20T18:35:34.132 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c5c2494ca442a56f1a277b51de1b8b1a422ded18' 2023-03-20T18:35:37.190 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-20T18:35:37.190 INFO:teuthology.packaging:ref: None 2023-03-20T18:35:37.190 INFO:teuthology.packaging:tag: None 2023-03-20T18:35:37.190 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-03-17-1235-quincy 2023-03-20T18:35:37.190 INFO:teuthology.packaging:sha1: c5c2494ca442a56f1a277b51de1b8b1a422ded18 2023-03-20T18:35:37.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri3-testing-2023-03-17-1235-quincy 2023-03-20T18:35:37.339 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-1352-gc5c2494c-1focal 2023-03-20T18:35:39.667 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-20T18:35:39.679 INFO:teuthology.task.internal:no buildpackages task found 2023-03-20T18:35:41.578 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-20T18:35:41.592 INFO:teuthology.task.internal:Saving configuration 2023-03-20T18:35:43.543 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-20T18:35:45.525 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-20T18:35:46.636 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-20 18:28:40.410570', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-03-20T18:35:48.618 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-20T18:35:48.640 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-03-20T18:35:50.604 INFO:teuthology.run_tasks:Running task console_log... 2023-03-20T18:35:50.656 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3b11a9ce50>, signals=[15]) 2023-03-20T18:35:53.398 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-20T18:35:53.411 INFO:teuthology.task.internal:Opening connections... 2023-03-20T18:35:55.562 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:35:55.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-20T18:35:56.269 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-20T18:35:57.739 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2023-03-20T18:35:59.413 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2023-03-20T18:36:00.751 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2023-03-20T18:36:00.757 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Ubuntu" 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:ID=ubuntu 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE=debian 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="20.04" 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.ubuntu.com/" 2023-03-20T18:36:01.894 INFO:teuthology.orchestra.run.smithi046.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-03-20T18:36:01.895 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-03-20T18:36:01.895 INFO:teuthology.orchestra.run.smithi046.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-03-20T18:36:01.895 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_CODENAME=focal 2023-03-20T18:36:01.895 INFO:teuthology.orchestra.run.smithi046.stdout:UBUNTU_CODENAME=focal 2023-03-20T18:36:01.896 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2023-03-20T18:36:01.919 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-20T18:36:03.221 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-20T18:36:04.383 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-20T18:36:06.910 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2023-03-20T18:36:06.916 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-20T18:36:09.583 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-20T18:36:12.299 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2023-03-20T18:36:12.318 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-20T18:36:15.639 INFO:teuthology.run_tasks:Running task kernel... 2023-03-20T18:36:16.948 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-03-20T18:36:17.972 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-03-20T18:36:17.972 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-20T18:36:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:36:19.485 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-03-20T18:36:19.491 INFO:teuthology.orchestra.run.smithi046.stdout:5.4.0-124-generic 2023-03-20T18:36:19.693 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.4.0-124-generic 2023-03-20T18:36:19.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2023-03-20T18:36:19.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2023-03-20T18:36:20.372 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-03-20T18:36:20.832 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-03-20T18:36:20.832 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-03-20T18:36:20.832 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-03-20T18:36:20.832 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [796 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,426 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [415 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.5 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,666 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [234 kB] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [620 B] 2023-03-20T18:36:21.004 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,038 kB] 2023-03-20T18:36:21.079 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [719 kB] 2023-03-20T18:36:21.598 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,046 kB] 2023-03-20T18:36:22.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [243 kB] 2023-03-20T18:36:22.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.8 kB] 2023-03-20T18:36:22.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.1 kB] 2023-03-20T18:36:22.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-03-20T18:36:22.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2023-03-20T18:36:22.179 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-03-20T18:36:22.179 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-03-20T18:36:22.179 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-03-20T18:36:22.179 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [568 kB] 2023-03-20T18:36:22.828 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [333 kB] 2023-03-20T18:36:22.828 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [12.3 kB] 2023-03-20T18:36:22.828 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,556 kB] 2023-03-20T18:36:22.829 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-03-20T18:36:22.829 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [219 kB] 2023-03-20T18:36:22.830 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [624 B] 2023-03-20T18:36:22.830 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [588 kB] 2023-03-20T18:36:22.830 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [812 kB] 2023-03-20T18:36:22.830 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [161 kB] 2023-03-20T18:36:22.830 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [17.2 kB] 2023-03-20T18:36:22.831 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.9 kB] 2023-03-20T18:36:22.831 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,488 B] 2023-03-20T18:36:22.831 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [528 B] 2023-03-20T18:36:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 14.4 MB in 2s (6,475 kB/s) 2023-03-20T18:36:25.958 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:36:28.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-03-20T18:36:28.303 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-5.4.0-144 linux-headers-5.4.0-144-generic 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-generic linux-image-5.4.0-144-generic 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-5.4.0-144-generic linux-modules-extra-5.4.0-144-generic 2023-03-20T18:36:29.352 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: thermald 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-5.4.0-144 linux-headers-5.4.0-144-generic 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-144-generic linux-modules-5.4.0-144-generic 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-extra-5.4.0-144-generic 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-generic linux-image-generic 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:3 upgraded, 5 newly installed, 0 to remove and 162 not upgraded. 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 77.1 MB of archives. 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 380 MB of additional disk space will be used. 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-144-generic amd64 5.4.0-144.161 [15.0 MB] 2023-03-20T18:36:29.353 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-144-generic amd64 5.4.0-144.161 [10.5 MB] 2023-03-20T18:36:30.547 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-144-generic amd64 5.4.0-144.161 [39.2 MB] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.144.142 [1,896 B] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.144.142 [2,484 B] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-144 all 5.4.0-144.161 [11.0 MB] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-144-generic amd64 5.4.0-144.161 [1,374 kB] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.144.142 [2,356 B] 2023-03-20T18:36:30.548 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 77.1 MB in 1s (78.6 MB/s) 2023-03-20T18:36:30.671 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-5.4.0-144-generic. 2023-03-20T18:36:32.354 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-03-20T18:36:32.918 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../0-linux-modules-5.4.0-144-generic_5.4.0-144.161_amd64.deb ... 2023-03-20T18:36:32.919 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:34.487 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-image-5.4.0-144-generic. 2023-03-20T18:36:35.666 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../1-linux-image-5.4.0-144-generic_5.4.0-144.161_amd64.deb ... 2023-03-20T18:36:37.539 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:37.540 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-144-generic. 2023-03-20T18:36:37.540 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-144-generic_5.4.0-144.161_amd64.deb ... 2023-03-20T18:36:37.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-extra-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:40.634 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../3-linux-generic_5.4.0.144.142_amd64.deb ... 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-generic (5.4.0.144.142) over (5.4.0.124.125) ... 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.144.142_amd64.deb ... 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-generic (5.4.0.144.142) over (5.4.0.124.125) ... 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.4.0-144. 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../5-linux-headers-5.4.0-144_5.4.0-144.161_all.deb ... 2023-03-20T18:36:43.774 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.4.0-144 (5.4.0-144.161) ... 2023-03-20T18:36:47.251 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.4.0-144-generic. 2023-03-20T18:36:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../6-linux-headers-5.4.0-144-generic_5.4.0-144.161_amd64.deb ... 2023-03-20T18:36:48.671 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:50.098 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.144.142_amd64.deb ... 2023-03-20T18:36:50.736 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-generic (5.4.0.144.142) over (5.4.0.124.125) ... 2023-03-20T18:36:50.736 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.4.0-144 (5.4.0-144.161) ... 2023-03-20T18:36:51.948 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:51.998 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:53.394 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-144-generic 2023-03-20T18:36:55.870 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-144-generic 2023-03-20T18:36:55.870 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-extra-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:55.871 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:55.871 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-generic (5.4.0.144.142) ... 2023-03-20T18:36:55.871 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-generic (5.4.0.144.142) ... 2023-03-20T18:36:55.872 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-generic (5.4.0.144.142) ... 2023-03-20T18:36:55.872 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for linux-image-5.4.0-144-generic (5.4.0-144.161) ... 2023-03-20T18:36:55.872 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-03-20T18:36:55.872 INFO:teuthology.orchestra.run.smithi046.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-144-generic 2023-03-20T18:37:17.681 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-03-20T18:37:19.698 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub' 2023-03-20T18:37:19.699 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-03-20T18:37:19.699 INFO:teuthology.orchestra.run.smithi046.stdout:Generating grub configuration file ... 2023-03-20T18:37:19.699 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-144-generic 2023-03-20T18:37:19.699 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-144-generic 2023-03-20T18:37:19.700 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-03-20T18:37:19.700 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-03-20T18:37:21.232 INFO:teuthology.orchestra.run.smithi046.stdout:done 2023-03-20T18:37:21.949 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2023-03-20T18:37:22.008 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 19 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2023-03-20T18:37:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.4.0.144.142 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.4.0-144-generic, linux-modules-extra-5.4.0-144-generic, linux-firmware, intel-microcode, amd64-microcode 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2023-03-20T18:37:23.491 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2023-03-20T18:37:23.492 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-144-generic 2023-03-20T18:37:23.492 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-03-20T18:37:23.492 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-03-20T18:37:23.493 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-03-20T18:37:23.493 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2023-03-20T18:37:23.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2023-03-20T18:37:23.571 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2023-03-20T18:37:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-03-20T18:37:28.497 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-03-20T18:37:28.499 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-03-20T18:37:28.499 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-03-20T18:37:28.499 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:37:28.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-03-20T18:37:28.576 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:37:30.225 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2023-03-20T18:37:30.225 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2023-03-20T18:37:30.225 INFO:teuthology.orchestra.run.smithi046.stdout:linux-image-generic is already the newest version (5.4.0.144.142). 2023-03-20T18:37:30.225 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 0 newly installed, 0 to remove and 162 not upgraded. 2023-03-20T18:37:30.225 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2023-03-20T18:37:30.246 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2023-03-20T18:37:31.441 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2023-03-20T18:37:32.417 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2023-03-20T18:37:33.433 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2023-03-20T18:37:33.433 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 19 2023-03-20T18:37:33.433 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2023-03-20T18:37:33.433 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2023-03-20T18:37:33.434 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2023-03-20T18:37:33.434 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.4.0.144.142 2023-03-20T18:37:33.434 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-03-20T18:37:33.434 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.4.0-144-generic, linux-modules-extra-5.4.0-144-generic, linux-firmware, intel-microcode, amd64-microcode 2023-03-20T18:37:34.627 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2023-03-20T18:37:34.627 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2023-03-20T18:37:34.627 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2023-03-20T18:37:34.627 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2023-03-20T18:37:36.575 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-03-20T18:37:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.8KhhHHTJ2Q 2023-03-20T18:37:38.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub/grub.cfg /tmp/tmp.8KhhHHTJ2Q 2023-03-20T18:37:38.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.8KhhHHTJ2Q 2023-03-20T18:37:39.961 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.8KhhHHTJ2Q is 9KB 2023-03-20T18:37:40.942 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.8KhhHHTJ2Q 2023-03-20T18:37:43.056 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-03-20T18:37:44.732 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:37:46.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-03-20T18:37:46.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-03-20T18:37:46.116 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-144-generic 2023-03-20T18:37:48.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo update-grub 2023-03-20T18:37:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-144-generic 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-144-generic 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-03-20T18:37:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-03-20T18:37:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-03-20T18:37:52.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2023-03-20T18:38:23.113 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-03-20T18:38:25.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-03-20T18:38:25.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-20T18:38:43.717 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-03-20T18:39:16.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-03-20T18:39:17.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-20T18:39:35.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-03-20T18:40:06.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-03-20T18:40:08.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-20T18:40:08.343 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-03-20T18:40:09.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-03-20T18:40:11.039 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-144-generic"... 2023-03-20T18:40:11.039 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-03-20T18:40:11.046 INFO:teuthology.orchestra.run.smithi046.stdout:5.4.0-144-generic 2023-03-20T18:40:12.063 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-144-generic vs 5.4.0-144-generic 2023-03-20T18:40:13.287 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-03-20T18:40:13.287 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-03-20T18:40:14.288 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-03-20T18:40:15.345 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-03-20T18:40:15.437 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2023-03-20T18:40:16.390 DEBUG:teuthology.parallel:result is None 2023-03-20T18:40:16.390 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-20T18:40:16.399 INFO:teuthology.task.internal:Creating test directory... 2023-03-20T18:40:17.465 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-20T18:40:17.473 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-20T18:40:19.199 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-20T18:40:20.849 INFO:teuthology.task.internal:Creating archive directory... 2023-03-20T18:40:22.709 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-20T18:40:22.729 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-20T18:40:24.128 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-20T18:40:25.880 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-20T18:40:25.912 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-20T18:40:27.919 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-20T18:40:27.920 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-20T18:40:27.927 INFO:teuthology.task.internal:Configuring sudo... 2023-03-20T18:40:30.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-20T18:40:30.052 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-20T18:40:32.491 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-20T18:40:33.906 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-20T18:40:33.913 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-20T18:40:37.073 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-20T18:40:38.871 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:40:40.403 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-20T18:40:40.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2023-03-20T18:40:43.045 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-20T18:40:43.338 INFO:teuthology.task.internal:Starting timer... 2023-03-20T18:40:44.391 INFO:teuthology.run_tasks:Running task pcp... 2023-03-20T18:40:44.401 INFO:teuthology.run_tasks:Running task selinux... 2023-03-20T18:40:45.032 DEBUG:teuthology.task.selinux:Excluding smithi046: OS 'ubuntu' does not support SELinux 2023-03-20T18:40:46.329 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-20T18:40:46.329 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-03-20T18:40:46.329 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-20T18:40:46.329 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-20T18:40:46.339 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-03-20T18:40:52.748 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-20T18:40:52.769 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-20T18:40:57.556 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-20T18:46:21.335 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com')] 2023-03-20T18:46:23.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-03-20T18:46:23.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-20T18:46:23.557 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-03-20T18:46:27.996 INFO:teuthology.run_tasks:Running task clock... 2023-03-20T18:46:30.202 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-20T18:46:32.259 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-20T18:46:32.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-20T18:46:32.294 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-03-20T18:46:34.024 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Command line: ntpd -gq 2023-03-20T18:46:34.025 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: proto: precision = 0.090 usec (-23) 2023-03-20T18:46:36.717 INFO:teuthology.orchestra.run.smithi046.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-03-20T18:46:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-03-20T18:46:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: restrict ::: KOD does nothing without LIMITED. 2023-03-20T18:46:36.718 INFO:teuthology.orchestra.run.smithi046.stderr:restrict ::: KOD does nothing without LIMITED. 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen and drop on 0 v6wildcard [::]:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen normally on 2 lo 127.0.0.1:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen normally on 3 ens1f0 172.21.15.46:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen normally on 4 lo [::1]:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15f8%4]:123 2023-03-20T18:46:36.719 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:32 ntpd[21437]: Listening on routing socket on fd #22 for interface updates 2023-03-20T18:46:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:33 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:33 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:39.582 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:35 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:36 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:39.585 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:37 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:40.632 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:40.632 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:38 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:39 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:40 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: can't open /var/log/ntpstats/rawstats.20230320: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-03-20T18:46:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: can't open /var/log/ntpstats/peerstats.20230320: Permission denied 2023-03-20T18:46:42.797 INFO:teuthology.orchestra.run.smithi046.stdout:20 Mar 18:46:41 ntpd[21437]: ntpd: time slew +0.001577 s 2023-03-20T18:46:42.797 INFO:teuthology.orchestra.run.smithi046.stdout:ntpd: time slew +0.001577s 2023-03-20T18:46:42.798 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-03-20T18:46:42.798 INFO:teuthology.orchestra.run.smithi046.stderr:20 Mar 18:46:41 ntpd[21437]: can't open /var/log/ntpstats/loopstats.20230320: Permission denied 2023-03-20T18:46:42.799 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2023-03-20T18:46:42.799 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2023-03-20T18:46:42.799 INFO:teuthology.orchestra.run.smithi046.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-03-20T18:46:42.799 INFO:teuthology.orchestra.run.smithi046.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-03-20T18:46:42.800 INFO:teuthology.orchestra.run.smithi046.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-03-20T18:46:42.800 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-03-20T18:46:42.800 INFO:teuthology.run_tasks:Running task install... 2023-03-20T18:46:46.822 DEBUG:teuthology.task.install:project ceph 2023-03-20T18:46:48.384 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18'}} 2023-03-20T18:46:48.384 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18'} 2023-03-20T18:46:48.384 INFO:teuthology.task.install:Using flavor: default 2023-03-20T18:46:48.389 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-20T18:46:50.579 INFO:teuthology.task.install:extra packages: [] 2023-03-20T18:46:50.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-key list | grep Ceph 2023-03-20T18:46:50.686 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-03-20T18:46:52.852 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-03-20T18:46:52.852 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph.com (release key) 2023-03-20T18:46:52.852 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-03-20T18:46:52.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c5c2494ca442a56f1a277b51de1b8b1a422ded18 2023-03-20T18:46:52.997 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default/ 2023-03-20T18:46:54.561 INFO:teuthology.task.install.deb:Package version is 17.2.5-1352-gc5c2494c-1focal 2023-03-20T18:46:54.690 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:46:56.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-03-20T18:46:56.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2023-03-20T18:46:58.210 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal InRelease 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal Release.gpg 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main i386 Packages [3,382 B] 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 28.4 kB in 1s (43.3 kB/s) 2023-03-20T18:47:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:47:00.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.5-1352-gc5c2494c-1focal cephadm=17.2.5-1352-gc5c2494c-1focal ceph-mds=17.2.5-1352-gc5c2494c-1focal ceph-mgr=17.2.5-1352-gc5c2494c-1focal ceph-common=17.2.5-1352-gc5c2494c-1focal ceph-fuse=17.2.5-1352-gc5c2494c-1focal ceph-test=17.2.5-1352-gc5c2494c-1focal ceph-volume=17.2.5-1352-gc5c2494c-1focal radosgw=17.2.5-1352-gc5c2494c-1focal python3-rados=17.2.5-1352-gc5c2494c-1focal python3-rgw=17.2.5-1352-gc5c2494c-1focal python3-cephfs=17.2.5-1352-gc5c2494c-1focal python3-rbd=17.2.5-1352-gc5c2494c-1focal libcephfs2=17.2.5-1352-gc5c2494c-1focal libcephfs-dev=17.2.5-1352-gc5c2494c-1focal librados2=17.2.5-1352-gc5c2494c-1focal librbd1=17.2.5-1352-gc5c2494c-1focal rbd-fuse=17.2.5-1352-gc5c2494c-1focal 2023-03-20T18:47:00.469 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-03-20T18:47:02.688 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2023-03-20T18:47:04.736 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2023-03-20T18:47:04.736 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2023-03-20T18:47:04.736 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-03-20T18:47:04.736 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2023-03-20T18:47:04.737 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2023-03-20T18:47:04.737 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-03-20T18:47:04.737 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-03-20T18:47:04.737 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-03-20T18:47:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-03-20T18:47:07.928 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: socat xmlstarlet 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-03-20T18:47:07.929 INFO:teuthology.orchestra.run.smithi046.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: btrfs-tools 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-03-20T18:47:07.930 INFO:teuthology.orchestra.run.smithi046.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-03-20T18:47:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 librbd1 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout:2 upgraded, 108 newly installed, 0 to remove and 152 not upgraded. 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 209 MB of archives. 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 978 MB of additional disk space will be used. 2023-03-20T18:47:09.409 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.5-1352-gc5c2494c-1focal [3,503 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-03-20T18:47:09.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-03-20T18:47:09.913 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-03-20T18:47:11.276 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.5-1352-gc5c2494c-1focal [3,551 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-03-20T18:47:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-03-20T18:47:13.435 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-03-20T18:47:13.435 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-03-20T18:47:13.435 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.5-1352-gc5c2494c-1focal [756 kB] 2023-03-20T18:47:13.435 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.5-1352-gc5c2494c-1focal [413 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.5-1352-gc5c2494c-1focal [22.5 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.5-1352-gc5c2494c-1focal [200 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.5-1352-gc5c2494c-1focal [45.5 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.5-1352-gc5c2494c-1focal [384 kB] 2023-03-20T18:47:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.5-1352-gc5c2494c-1focal [5,554 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-03-20T18:47:14.490 INFO:teuthology.orchestra.run.smithi046.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.5-1352-gc5c2494c-1focal [116 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.5-1352-gc5c2494c-1focal [479 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.5-1352-gc5c2494c-1focal [27.2 MB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-03-20T18:47:14.491 INFO:teuthology.orchestra.run.smithi046.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-03-20T18:47:15.401 INFO:teuthology.orchestra.run.smithi046.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-03-20T18:47:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-03-20T18:47:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-03-20T18:47:15.408 INFO:teuthology.orchestra.run.smithi046.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.5-1352-gc5c2494c-1focal [6,440 kB] 2023-03-20T18:47:18.411 INFO:teuthology.orchestra.run.smithi046.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.5-1352-gc5c2494c-1focal [219 kB] 2023-03-20T18:47:20.913 INFO:teuthology.orchestra.run.smithi046.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.5-1352-gc5c2494c-1focal [152 kB] 2023-03-20T18:47:20.913 INFO:teuthology.orchestra.run.smithi046.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.5-1352-gc5c2494c-1focal [1,584 kB] 2023-03-20T18:47:20.913 INFO:teuthology.orchestra.run.smithi046.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.5-1352-gc5c2494c-1focal [7,830 kB] 2023-03-20T18:47:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.5-1352-gc5c2494c-1focal [28.6 MB] 2023-03-20T18:47:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.5-1352-gc5c2494c-1focal [3,908 B] 2023-03-20T18:47:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.5-1352-gc5c2494c-1focal [866 kB] 2023-03-20T18:47:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.5-1352-gc5c2494c-1focal [2,394 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.5-1352-gc5c2494c-1focal [78.4 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.5-1352-gc5c2494c-1focal [113 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.5-1352-gc5c2494c-1focal [1,720 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.5-1352-gc5c2494c-1focal [7,877 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.5-1352-gc5c2494c-1focal [4,024 B] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.5-1352-gc5c2494c-1focal [56.1 MB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.5-1352-gc5c2494c-1focal [225 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.5-1352-gc5c2494c-1focal [21.8 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.5-1352-gc5c2494c-1focal [17.0 MB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-03-17-1235-quincy/c5c2494ca442a56f1a277b51de1b8b1a422ded18/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.5-1352-gc5c2494c-1focal [79.4 kB] 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 209 MB in 6s (33.9 MB/s) 2023-03-20T18:47:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141756 files and directories currently installed.) 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../001-librbd1_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librbd1 (17.2.5-1352-gc5c2494c-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../002-librados2_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librados2 (17.2.5-1352-gc5c2494c-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs2. 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../003-libcephfs2_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs2 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rados. 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../004-python3-rados_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rados (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.385 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-argparse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cephfs. 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../006-python3-cephfs_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cephfs (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-common. 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-common (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-prettytable. 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rbd. 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../009-python3-rbd_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rbd (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:22.387 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-03-20T18:47:22.387 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-03-20T18:47:22.387 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-03-20T18:47:23.984 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-03-20T18:47:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-03-20T18:47:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-03-20T18:47:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-03-20T18:47:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-03-20T18:47:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:47:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-03-20T18:47:25.503 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-03-20T18:47:25.503 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:47:25.833 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-03-20T18:47:25.833 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-03-20T18:47:25.833 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:47:25.833 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-03-20T18:47:26.219 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-03-20T18:47:27.171 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-03-20T18:47:27.171 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librgw2. 2023-03-20T18:47:27.171 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../019-librgw2_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:27.171 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librgw2 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:27.171 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rgw. 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../020-python3-rgw_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rgw (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liboath0. 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-03-20T18:47:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libradosstriper1. 2023-03-20T18:47:27.345 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../022-libradosstriper1_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:27.345 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libradosstriper1 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:27.760 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-common. 2023-03-20T18:47:28.589 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../023-ceph-common_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:28.589 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-common (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:28.589 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-base. 2023-03-20T18:47:28.589 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../024-ceph-base_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-base (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cryptography. 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-openssl. 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-03-20T18:47:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-modules-core (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bcrypt. 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cherrypy3. 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-03-20T18:47:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-03-20T18:47:29.726 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-natsort. 2023-03-20T18:47:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-03-20T18:47:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-03-20T18:47:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jwt. 2023-03-20T18:47:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-logutils. 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-markupsafe. 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-mako. 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplegeneric. 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-03-20T18:47:31.540 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-singledispatch. 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webob. 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-soupsieve. 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bs4. 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-waitress. 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-03-20T18:47:31.541 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempita. 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-paste. 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-03-20T18:47:31.542 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-03-20T18:47:31.543 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-03-20T18:47:32.986 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-03-20T18:47:32.986 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-03-20T18:47:32.986 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastedeploy. 2023-03-20T18:47:32.986 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-03-20T18:47:32.986 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-03-20T18:47:34.810 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webtest. 2023-03-20T18:47:34.810 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-03-20T18:47:34.810 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-03-20T18:47:34.810 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pecan. 2023-03-20T18:47:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-03-20T18:47:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-werkzeug. 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libsqlite3-mod-ceph (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr. 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../049-ceph-mgr_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:36.058 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mon. 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../050-ceph-mon_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mon (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-osd. 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../051-ceph-osd_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:37.072 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-osd (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:37.863 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph. 2023-03-20T18:47:39.751 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../052-ceph_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-fuse. 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../053-ceph-fuse_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-fuse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mds. 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../054-ceph-mds_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:39.752 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mds (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:41.133 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephadm. 2023-03-20T18:47:41.133 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../055-cephadm_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:41.133 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephadm (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:41.134 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jinja2. 2023-03-20T18:47:41.135 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-03-20T18:47:41.135 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-03-20T18:47:41.135 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-03-20T18:47:41.135 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-cephadm (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-repoze.lru. 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-underscore. 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-03-20T18:47:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-03-20T18:47:41.268 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-03-20T18:47:42.603 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-03-20T18:47:43.773 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-routes. 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:44.100 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-dashboard (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:44.101 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-decorator. 2023-03-20T18:47:44.101 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-scipy. 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-03-20T18:47:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-03-20T18:47:44.816 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-03-20T18:47:45.291 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-03-20T18:47:45.291 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-03-20T18:47:45.444 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-joblib. 2023-03-20T18:47:46.499 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-03-20T18:47:46.499 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-03-20T18:47:46.499 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn. 2023-03-20T18:47:46.499 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-03-20T18:47:46.499 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-03-20T18:47:46.722 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-03-20T18:47:47.858 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:47.858 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cachetools. 2023-03-20T18:47:49.127 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-03-20T18:47:49.128 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-03-20T18:47:49.128 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1. 2023-03-20T18:47:49.128 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-03-20T18:47:49.128 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-03-20T18:47:49.128 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-03-20T18:47:50.788 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-03-20T18:47:50.788 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-03-20T18:47:50.788 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rsa. 2023-03-20T18:47:50.788 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rsa (4.0-3) ... 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-google-auth. 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-blinker. 2023-03-20T18:47:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-oauthlib. 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-03-20T18:47:50.790 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-03-20T18:47:51.030 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-websocket. 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-kubernetes. 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-k8sevents (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:52.559 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libonig5:amd64. 2023-03-20T18:47:53.700 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-03-20T18:47:53.701 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-03-20T18:47:53.701 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjq1:amd64. 2023-03-20T18:47:56.032 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-03-20T18:47:56.032 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-03-20T18:47:56.032 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package jq. 2023-03-20T18:47:56.032 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-03-20T18:47:56.032 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-03-20T18:47:56.033 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package socat. 2023-03-20T18:47:56.033 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-03-20T18:47:56.033 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking socat (1.7.3.3-2) ... 2023-03-20T18:47:58.125 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package xmlstarlet. 2023-03-20T18:47:58.125 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-03-20T18:47:58.125 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-03-20T18:47:58.125 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-test. 2023-03-20T18:47:58.125 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../086-ceph-test_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:47:59.649 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-test (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:47:59.650 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-volume. 2023-03-20T18:47:59.650 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../087-ceph-volume_17.2.5-1352-gc5c2494c-1focal_all.deb ... 2023-03-20T18:47:59.678 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-volume (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:00.851 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package fonts-mathjax. 2023-03-20T18:48:00.851 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-03-20T18:48:00.852 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-03-20T18:48:00.852 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs-dev. 2023-03-20T18:48:00.852 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:48:00.852 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs-dev (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:01.105 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package nvme-cli. 2023-03-20T18:48:02.931 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-03-20T18:48:02.931 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-03-20T18:48:02.931 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-mathjax. 2023-03-20T18:48:02.931 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-03-20T18:48:02.931 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-03-20T18:48:03.771 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-natsort-doc. 2023-03-20T18:48:06.193 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-03-20T18:48:06.193 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-03-20T18:48:06.194 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-atomicwrites. 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-attr. 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webencodings. 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-html5lib. 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-03-20T18:48:08.452 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-03-20T18:48:08.453 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-03-20T18:48:08.454 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-packaging. 2023-03-20T18:48:08.454 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-03-20T18:48:08.454 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-packaging (20.3-1) ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastescript. 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pluggy. 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-psutil. 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-py. 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyinotify. 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-03-20T18:48:08.455 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-03-20T18:48:08.694 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-wcwidth. 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pytest. 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplejson. 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-03-20T18:48:10.518 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-03-20T18:48:10.751 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package radosgw. 2023-03-20T18:48:13.933 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../108-radosgw_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:48:13.933 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking radosgw (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package rbd-fuse. 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../109-rbd-fuse_17.2.5-1352-gc5c2494c-1focal_amd64.deb ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking rbd-fuse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-attr (19.3.0-2) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-03-20T18:48:13.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephadm (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:14.433 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user cephadm....done 2023-03-20T18:48:15.939 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-03-20T18:48:17.456 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-03-20T18:48:17.456 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-03-20T18:48:17.456 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-03-20T18:48:17.457 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-03-20T18:48:17.457 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-03-20T18:48:17.457 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-03-20T18:48:17.457 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-03-20T18:48:17.458 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-03-20T18:48:17.458 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-03-20T18:48:17.458 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-03-20T18:48:17.642 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-argparse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:18.564 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-03-20T18:48:18.564 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-03-20T18:48:20.717 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-03-20T18:48:20.718 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-03-20T18:48:20.718 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-03-20T18:48:20.718 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-03-20T18:48:20.719 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-03-20T18:48:20.719 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-03-20T18:48:20.719 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-03-20T18:48:20.719 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-03-20T18:48:20.756 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-03-20T18:48:23.253 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-packaging (20.3-1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-03-20T18:48:26.694 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up socat (1.7.3.3-2) ... 2023-03-20T18:48:26.695 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-common (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-03-20T18:48:28.502 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:48:29.932 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-03-20T18:48:37.304 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-03-20T18:48:37.307 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-03-20T18:48:40.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-03-20T18:48:40.376 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-03-20T18:48:40.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-03-20T18:48:40.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-03-20T18:48:40.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librados2 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libsqlite3-mod-ceph (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-03-20T18:48:43.134 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs2 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libradosstriper1 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librbd1 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:43.135 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-03-20T18:48:43.136 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-03-20T18:48:43.136 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-03-20T18:48:43.136 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rsa (4.0-3) ... 2023-03-20T18:48:47.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-03-20T18:48:47.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-03-20T18:48:47.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-modules-core (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-fuse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.749 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-03-20T18:48:47.750 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-03-20T18:48:47.750 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs-dev (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.752 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rados (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librgw2 (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rbd (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up rbd-fuse (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-routes (2.4.1-2) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rgw (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cephfs (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-03-20T18:48:47.753 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-common (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:47.754 INFO:teuthology.orchestra.run.smithi046.stdout:Adding group ceph....done 2023-03-20T18:48:47.754 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user ceph....done 2023-03-20T18:48:47.754 INFO:teuthology.orchestra.run.smithi046.stdout:Setting system user ceph properties....done 2023-03-20T18:48:47.754 INFO:teuthology.orchestra.run.smithi046.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-03-20T18:48:49.367 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-test (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up radosgw (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-base (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mds (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-03-20T18:48:49.368 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-osd (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.436 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-03-20T18:48:52.436 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-03-20T18:48:52.436 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-k8sevents (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.436 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-diskprediction-local (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.436 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mon (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.437 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-03-20T18:48:52.437 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-03-20T18:48:52.438 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-cephadm (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.438 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.438 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-dashboard (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-volume (17.2.5-1352-gc5c2494c-1focal) ... 2023-03-20T18:48:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-03-20T18:48:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-03-20T18:48:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-03-20T18:48:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-03-20T18:48:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-03-20T18:48:54.894 DEBUG:teuthology.parallel:result is None 2023-03-20T18:48:54.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c5c2494ca442a56f1a277b51de1b8b1a422ded18 2023-03-20T18:48:56.563 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-query -W -f '${Version}' ceph 2023-03-20T18:48:58.188 INFO:teuthology.orchestra.run.smithi046.stdout:17.2.5-1352-gc5c2494c-1focal 2023-03-20T18:48:59.706 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-1352-gc5c2494c-1focal 2023-03-20T18:48:59.707 INFO:teuthology.task.install:The correct ceph version 17.2.5-1352-gc5c2494c-1focal is installed. 2023-03-20T18:48:59.707 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-20T18:48:59.707 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:48:59.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-20T18:48:59.725 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-20T18:49:00.545 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:49:01.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2023-03-20T18:49:01.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-20T18:49:02.658 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-20T18:49:04.520 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:49:04.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-20T18:49:04.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-20T18:49:06.979 INFO:teuthology.run_tasks:Running task ceph... 2023-03-20T18:49:09.737 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-20T18:49:12.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2023-03-20T18:49:12.291 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-20T18:49:13.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-20T18:49:13.776 INFO:tasks.ceph:Creating extra log directories... 2023-03-20T18:49:14.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-20T18:49:14.658 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-20T18:49:19.654 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2023-03-20T18:49:19.654 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146', 'branch': 'wip-yuri3-testing-2023-03-17-1235-quincy', 'description': 'rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}', 'email': None, 'first_in_suite': False, 'job_id': '7213146', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-03-17-1235-quincy'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a6dd66830c12498d02f3894e352dd8b49a7bab4c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a6dd66830c12498d02f3894e352dd8b49a7bab4c', 'targets': {'smithi046.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': 'c5c2494ca442a56f1a277b51de1b8b1a422ded18', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool', 'mon/pool_ops.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '7f7f7c33a372789d0252bbbcd5798300b0f044d0', 'timestamp': '2023-03-18_00:57:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4737'} 2023-03-20T18:49:19.655 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-20T18:49:19.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2023-03-20T18:49:23.380 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:49:24.830 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2023-03-20T18:49:24.836 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-20T18:49:27.866 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2023-03-20T18:49:27.875 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-03-20 18:44:48.976826336 +0000 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-03-20 18:44:48.972826427 +0000 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-03-20 18:44:48.972826427 +0000 2023-03-20T18:49:30.210 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-03-20T18:49:30.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-20T18:49:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-03-20T18:49:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-03-20T18:49:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000299873 s, 1.7 MB/s 2023-03-20T18:49:33.322 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-20T18:49:33.378 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2023-03-20T18:49:35.040 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-03-20 18:44:49.792807556 +0000 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-03-20 18:44:49.488814554 +0000 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-03-20 18:44:49.488814554 +0000 2023-03-20T18:49:36.424 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-03-20T18:49:36.424 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-20T18:49:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-03-20T18:49:37.593 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-03-20T18:49:37.593 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.00030043 s, 1.7 MB/s 2023-03-20T18:49:37.595 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-20T18:49:38.382 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2023-03-20T18:49:39.449 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-20T18:49:40.358 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-20T18:49:40.358 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-03-20T18:49:40.358 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-20T18:49:40.358 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-03-20 18:44:50.296795958 +0000 2023-03-20T18:49:40.358 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-03-20 18:44:50.024802218 +0000 2023-03-20T18:49:40.359 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-03-20 18:44:50.024802218 +0000 2023-03-20T18:49:40.359 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-03-20T18:49:40.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-20T18:49:40.374 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-03-20T18:49:41.731 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-03-20T18:49:41.731 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000301554 s, 1.7 MB/s 2023-03-20T18:49:41.732 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-20T18:49:41.742 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2023-03-20T18:49:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-20T18:49:43.715 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-20T18:49:43.715 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-03-20T18:49:44.955 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-20T18:49:44.955 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-03-20 18:44:50.804784269 +0000 2023-03-20T18:49:44.955 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-03-20 18:44:50.512790989 +0000 2023-03-20T18:49:44.956 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-03-20 18:44:50.512790989 +0000 2023-03-20T18:49:46.173 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-03-20T18:49:46.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-20T18:49:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-03-20T18:49:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-03-20T18:49:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000317456 s, 1.6 MB/s 2023-03-20T18:49:47.552 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-20T18:49:47.602 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-03-20T18:49:49.599 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-03-20T18:49:49.599 INFO:tasks.ceph:Generating config... 2023-03-20T18:49:49.601 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-03-20T18:49:49.601 INFO:tasks.ceph:[global] ms bind msgr2 = False 2023-03-20T18:49:49.601 INFO:tasks.ceph:[global] ms type = async 2023-03-20T18:49:49.601 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-20T18:49:49.602 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-20T18:49:49.602 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-20T18:49:49.602 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-20T18:49:49.602 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-20T18:49:51.106 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-03-20T18:49:52.834 INFO:tasks.ceph:[osd] bdev async discard = True 2023-03-20T18:49:54.141 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-03-20T18:49:54.141 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-03-20T18:49:54.141 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-20T18:49:54.141 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-20T18:49:54.142 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-03-20T18:49:54.142 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-20T18:49:55.798 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-20T18:49:55.799 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-20T18:49:55.799 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-03-20T18:49:55.799 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-03-20T18:49:55.799 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-20T18:49:57.340 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-03-20T18:49:57.340 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-20T18:49:57.340 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-03-20T18:49:57.340 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-03-20T18:49:57.340 INFO:tasks.ceph:Setting up mon.a... 2023-03-20T18:49:57.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-20T18:49:57.378 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2023-03-20T18:49:59.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-20T18:49:59.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-20T18:50:00.880 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.46:6789]'), ('mon.b', '[v1:172.21.15.46:6790]'), ('mon.c', '[v1:172.21.15.46:6791]')] 2023-03-20T18:50:02.273 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.46:6789],[v1:172.21.15.46:6790],[v1:172.21.15.46:6791]', 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-03-20T18:50:02.275 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:50:03.948 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-20T18:50:03.967 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.46:6789]' --addv b '[v1:172.21.15.46:6790]' --addv c '[v1:172.21.15.46:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:50:05.919 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:50:07.704 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 45227b9a-cda4-41c5-b809-1315c0145b2a 2023-03-20T18:50:10.034 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = octopus 2023-03-20T18:50:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2023-03-20T18:50:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 45227b9a-cda4-41c5-b809-1315c0145b2a 2023-03-20T18:50:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2023-03-20T18:50:05.916746+0000 2023-03-20T18:50:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:created 2023-03-20T18:50:05.916746+0000 2023-03-20T18:50:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 15 (octopus) 2023-03-20T18:50:11.564 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2023-03-20T18:50:11.564 INFO:teuthology.orchestra.run.smithi046.stdout:0: v1:172.21.15.46:6789/0 mon.a 2023-03-20T18:50:11.564 INFO:teuthology.orchestra.run.smithi046.stdout:1: v1:172.21.15.46:6790/0 mon.b 2023-03-20T18:50:11.565 INFO:teuthology.orchestra.run.smithi046.stdout:2: v1:172.21.15.46:6791/0 mon.c 2023-03-20T18:50:11.565 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-20T18:50:11.566 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-20T18:50:11.572 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 45227b9a-cda4-41c5-b809-1315c0145b2a... 2023-03-20T18:50:13.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-20T18:50:15.208 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2023-03-20T18:50:16.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2023-03-20T18:50:16.271 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2023-03-20T18:50:16.272 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-20T18:50:16.272 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-20T18:50:16.272 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2023-03-20T18:50:16.272 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:16.272 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-20T18:50:17.635 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:17.635 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2023-03-20T18:50:17.636 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:17.636 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2023-03-20T18:50:17.636 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:17.636 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2023-03-20T18:50:17.636 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2023-03-20T18:50:18.659 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "[v1:172.21.15.46:6789],[v1:172.21.15.46:6790],[v1:172.21.15.46:6791]" 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: mon election default strategy = 1 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: ms bind msgr2 = False 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: ms type = async 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 45227b9a-cda4-41c5-b809-1315c0145b2a 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.660 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2023-03-20T18:50:18.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = bitmap 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore zero block detection = True 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2023-03-20T18:50:18.662 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2023-03-20T18:50:18.663 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2023-03-20T18:50:18.663 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2023-03-20T18:50:18.663 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify cached snaps = True 2023-03-20T18:50:19.768 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify missing on start = True 2023-03-20T18:50:19.768 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock override recovery settings = True 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue = debug_random 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue cut off = debug_random 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2023-03-20T18:50:19.769 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:23.801 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2023-03-20T18:50:26.648 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-03-20T18:50:26.649 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-20T18:50:27.876 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:28.899 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub interval = 300 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-20T18:50:30.079 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-20T18:50:30.689 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2023-03-20T18:50:30.689 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2023-03-20T18:50:30.689 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2023-03-20T18:50:30.690 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-20T18:50:30.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-20T18:50:30.780 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-20T18:50:32.433 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:50:32.433 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-20T18:50:32.440 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:50:33.573 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-20T18:50:34.560 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:50:36.963 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:50:36.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-20T18:50:36.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-20T18:50:36.990 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:50:39.035 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:50:39.042 INFO:tasks.ceph:Setting up mon nodes... 2023-03-20T18:50:42.095 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-20T18:50:42.095 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-20T18:50:42.142 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-20T18:50:45.514 INFO:tasks.ceph:Setting up mds nodes... 2023-03-20T18:50:45.514 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-20T18:50:45.514 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-20T18:50:45.551 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-20T18:50:46.738 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-20T18:50:46.738 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-03-20T18:50:46.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-20T18:50:46.755 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-03-20T18:50:47.665 INFO:tasks.ceph:role: osd.0 2023-03-20T18:50:47.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:50:47.665 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-20T18:50:47.709 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-20T18:50:49.122 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-20T18:50:49.123 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-20T18:50:49.123 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-20T18:50:49.124 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-03-20T18:50:50.301 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-20T18:50:50.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-20T18:50:51.135 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-03-20T18:50:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:50:51.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-20T18:50:51.986 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-03-20T18:50:53.436 INFO:tasks.ceph:role: osd.1 2023-03-20T18:50:53.436 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:50:53.436 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-20T18:50:53.480 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-20T18:50:55.036 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-20T18:50:55.036 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-20T18:50:55.036 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 2023-03-20T18:50:55.036 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-20T18:50:55.036 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-03-20T18:50:56.379 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-20T18:50:56.379 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-20T18:50:56.380 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-20T18:50:56.380 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-20T18:50:56.380 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-03-20T18:50:59.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-20T18:50:59.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-20T18:51:01.305 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-03-20T18:51:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:51:05.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-20T18:51:05.577 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-03-20T18:51:07.705 INFO:tasks.ceph:role: osd.2 2023-03-20T18:51:07.705 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:51:07.705 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-20T18:51:07.754 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-20T18:51:09.785 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-20T18:51:09.786 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-20T18:51:09.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-03-20T18:51:10.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-20T18:51:10.312 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-20T18:51:11.325 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-03-20T18:51:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:51:12.956 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:51:13.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-20T18:51:14.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-20T18:51:14.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-20T18:51:14.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-20T18:51:14.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-20T18:51:14.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.026+0000 7ff1e1fe13c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-20T18:51:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:13.594+0000 7ff1e1fe13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:14.378+0000 7ff1e1fe13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:15.170+0000 7ff1e1fe13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:16.311 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-20T18:51:20.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-20T18:51:25.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-20T18:51:25.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-20T18:51:25.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-20T18:51:25.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-20T18:51:25.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:22.674+0000 7f95144f13c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-20T18:51:25.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:23.238+0000 7f95144f13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:25.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:24.026+0000 7f95144f13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:25.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:24.810+0000 7f95144f13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:25.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-20T18:51:28.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:51:30.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-20T18:51:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-20T18:51:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-20T18:51:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-20T18:51:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-20T18:51:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:30.862+0000 7f7f38d6b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-20T18:51:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:31.422+0000 7f7f38d6b3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:35.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:32.202+0000 7f7f38d6b3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:35.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-03-20T18:51:32.990+0000 7f7f38d6b3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-03-20T18:51:35.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-20T18:51:35.723 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-20T18:51:38.056 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:51:40.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-20T18:51:43.281 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:51:45.497 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-20T18:51:45.516 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:51:48.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-20T18:51:48.699 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:51:51.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-20T18:51:51.716 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:51:53.867 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-20T18:51:53.876 INFO:tasks.ceph:Adding keys to all mons... 2023-03-20T18:51:56.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-20T18:51:56.581 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQACqxhkwtFYCBAAblknSEd5QC8MCwJDeeWHWw== 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAhqxhkU47OARAA+eSdDPBd9OYzRNjT4KQL6Q== 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAqqxhkP7ROKBAAd1n7gc3BJovRmovGdFYcFQ== 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAyqxhknTCWMxAApLBkneUCfQno+VJpr98auA== 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-03-20T18:51:57.826 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAFqxhkLYezIBAAKkONExYT/NnVa1b8pHMJNA== 2023-03-20T18:51:57.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-20T18:51:57.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-20T18:51:59.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-20T18:52:03.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-20T18:52:04.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-20T18:52:06.599 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-20T18:52:08.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-20T18:52:08.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-20T18:52:09.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-20T18:52:11.571 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-20T18:52:12.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-20T18:52:13.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-20T18:52:14.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-20T18:52:15.796 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-20T18:52:17.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-20T18:52:19.459 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-20T18:52:21.243 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-20T18:52:22.897 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-20T18:52:22.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-20T18:52:22.900 INFO:tasks.ceph.mon.a:Started 2023-03-20T18:52:22.900 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-20T18:52:22.901 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-20T18:52:22.903 INFO:tasks.ceph.mon.b:Started 2023-03-20T18:52:24.213 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-20T18:52:24.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-20T18:52:24.217 INFO:tasks.ceph.mon.c:Started 2023-03-20T18:52:26.422 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-20T18:52:26.422 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-20T18:52:26.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-20T18:52:26.427 INFO:tasks.ceph.mgr.x:Started 2023-03-20T18:52:27.509 INFO:tasks.ceph:Setting crush tunables to default 2023-03-20T18:52:28.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2023-03-20T18:52:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2023-03-20T18:52:35.883 INFO:tasks.ceph:check_enable_crimson: False 2023-03-20T18:52:35.883 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-20T18:52:35.883 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:52:35.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-20T18:52:35.884 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:33.476+0000 7f46b1fe4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-20T18:52:37.109 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.120+0000 7f46b1fe4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-20T18:52:37.109 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.180+0000 7f46b1fe4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-20T18:52:37.110 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.296+0000 7f46b1fe4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-20T18:52:37.110 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.540+0000 7f46b1fe4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-20T18:52:37.110 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.592+0000 7f46b1fe4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.796+0000 7f46b1fe4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.864+0000 7f46b1fe4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:34.932+0000 7f46b1fe4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:35.376+0000 7f46b1fe4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:35.500+0000 7f46b1fe4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-20T18:52:38.293 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:35.732+0000 7f46b1fe4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-20T18:52:38.294 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.004+0000 7f46b1fe4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.072+0000 7f46b1fe4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.128+0000 7f46b1fe4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.184+0000 7f46b1fe4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.368+0000 7f46b1fe4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.544+0000 7f46b1fe4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.612+0000 7f46b1fe4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.776+0000 7f46b1fe4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:36.844+0000 7f46b1fe4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-20T18:52:38.295 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:37.572+0000 7f46b1fe4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-20T18:52:38.308 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:52:39.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-20T18:52:39.160 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:38.612+0000 7f469b121700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-20T18:52:39.161 INFO:tasks.ceph.mgr.x.smithi046.stderr:2023-03-20T18:52:38.612+0000 7f469b121700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-20T18:52:39.177 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:52:40.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-20T18:52:40.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new d7dbd9b2-81f9-46ac-bbad-beae282f1a88 0 2023-03-20T18:52:42.126 INFO:teuthology.orchestra.run.smithi046.stdout:0 2023-03-20T18:52:43.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 2b8049e1-db98-4e68-84e1-56c507402b29 1 2023-03-20T18:52:43.487 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-03-20T18:52:44.477 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 290b5c77-abaa-4e65-ab57-751368bbed68 2 2023-03-20T18:52:44.800 INFO:teuthology.orchestra.run.smithi046.stdout:2 2023-03-20T18:52:45.789 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-20T18:52:47.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-20T18:52:47.096 INFO:tasks.ceph.osd.0:Started 2023-03-20T18:52:48.319 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-20T18:52:48.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-20T18:52:50.266 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-03-20T18:52:47.696+0000 7f381b4013c0 -1 Falling back to public interface 2023-03-20T18:52:52.269 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-03-20T18:52:51.420+0000 7f381b4013c0 -1 osd.0 0 log_to_monitors true 2023-03-20T18:52:54.042 INFO:tasks.ceph.osd.1:Started 2023-03-20T18:52:54.042 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-20T18:52:54.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-20T18:52:54.043 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-03-20T18:52:52.504+0000 7f3816ba1700 -1 osd.0 0 waiting for initial osdmap 2023-03-20T18:52:54.043 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-03-20T18:52:52.508+0000 7f381314c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-20T18:52:54.043 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-03-20T18:52:52.872+0000 7f822ea3e3c0 -1 Falling back to public interface 2023-03-20T18:52:54.045 INFO:tasks.ceph.osd.2:Started 2023-03-20T18:52:54.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-20T18:52:54.334 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:52:56.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:52:53.502169+0000","last_up_change":"2023-03-20T18:52:53.502169+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:52:56.596 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-20T18:52:56.596 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-20T18:52:56.596 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-03-20T18:52:54.660+0000 7fcd2d1cb3c0 -1 Falling back to public interface 2023-03-20T18:52:56.597 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-03-20T18:52:56.584+0000 7f822ea3e3c0 -1 osd.1 0 log_to_monitors true 2023-03-20T18:52:56.898 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-20T18:52:58.170 INFO:teuthology.misc.health.smithi046.stdout: 2023-03-20T18:52:59.666 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:52:57.535363+0000","last_up_change":"2023-03-20T18:52:53.502169+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:52.453137+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:53:00.251 DEBUG:teuthology.misc:1 of 3 OSDs are up 2023-03-20T18:53:02.090 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-03-20T18:52:58.424+0000 7fcd2d1cb3c0 -1 osd.2 0 log_to_monitors true 2023-03-20T18:53:04.981 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-03-20T18:52:58.556+0000 7f822a1de700 -1 osd.1 0 waiting for initial osdmap 2023-03-20T18:53:04.981 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-03-20T18:52:58.564+0000 7f8222f82700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-20T18:53:04.982 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-03-20T18:52:59.564+0000 7fcd2896b700 -1 osd.2 0 waiting for initial osdmap 2023-03-20T18:53:04.982 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-03-20T18:52:59.568+0000 7fcd23713700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-20T18:53:08.391 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-20T18:53:10.236 INFO:teuthology.misc.health.smithi046.stdout: 2023-03-20T18:53:10.815 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":15,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:53:02.587282+0000","last_up_change":"2023-03-20T18:53:00.564459+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-20T18:52:59.675326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":29810}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":29810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":29810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":29810}]},"public_addr":"172.21.15.46:6805/29810","cluster_addr":"172.21.15.46:6806/29810","heartbeat_back_addr":"172.21.15.46:6808/29810","heartbeat_front_addr":"172.21.15.46:6807/29810","state":["exists","up"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":29850}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":29850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6812","nonce":29850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":29850}]},"public_addr":"172.21.15.46:6809/29850","cluster_addr":"172.21.15.46:6810/29850","heartbeat_back_addr":"172.21.15.46:6812/29850","heartbeat_front_addr":"172.21.15.46:6811/29850","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:52.453137+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:57.583791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:59.409344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:53:11.629 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-03-20T18:53:11.629 INFO:tasks.ceph:Creating RBD pool 2023-03-20T18:53:11.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-20T18:53:12.032 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'rbd' created 2023-03-20T18:53:13.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-20T18:53:14.049 INFO:teuthology.orchestra.run.smithi046.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-20T18:53:16.124 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-20T18:53:16.124 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-20T18:53:19.620 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-03-20T18:53:19.620 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-03-20T18:53:19.621 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-03-20T18:53:19.954 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:21.541 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6800","nonce":28755}]},"active_addr":"172.21.15.46:6800/28755","active_change":"2023-03-20T18:52:37.597535+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3783176197}]},{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1131018032}]},{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3640764180}]},{"addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3883465435}]}]} 2023-03-20T18:53:22.384 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-03-20T18:53:22.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-03-20T18:53:22.385 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-20T18:53:22.682 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:24.118 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:53:14.025668+0000","last_up_change":"2023-03-20T18:53:00.564459+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-20T18:52:59.675326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-20T18:53:11.952974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":29810}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":29810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":29810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":29810}]},"public_addr":"172.21.15.46:6805/29810","cluster_addr":"172.21.15.46:6806/29810","heartbeat_back_addr":"172.21.15.46:6808/29810","heartbeat_front_addr":"172.21.15.46:6807/29810","state":["exists","up"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":29850}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":29850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6812","nonce":29850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":29850}]},"public_addr":"172.21.15.46:6809/29850","cluster_addr":"172.21.15.46:6810/29850","heartbeat_back_addr":"172.21.15.46:6812/29850","heartbeat_front_addr":"172.21.15.46:6811/29850","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:52.453137+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:57.583791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:59.409344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:53:24.121 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-03-20T18:53:24.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-20T18:53:24.415 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:27.098 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:53:14.025668+0000","last_up_change":"2023-03-20T18:53:00.564459+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-20T18:52:59.675326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-20T18:53:11.952974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":29810}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":29810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":29810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":29810}]},"public_addr":"172.21.15.46:6805/29810","cluster_addr":"172.21.15.46:6806/29810","heartbeat_back_addr":"172.21.15.46:6808/29810","heartbeat_front_addr":"172.21.15.46:6807/29810","state":["exists","up"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":29850}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":29850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6812","nonce":29850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":29850}]},"public_addr":"172.21.15.46:6809/29850","cluster_addr":"172.21.15.46:6810/29850","heartbeat_back_addr":"172.21.15.46:6812/29850","heartbeat_front_addr":"172.21.15.46:6811/29850","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:52.453137+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:57.583791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:59.409344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:53:27.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-03-20T18:53:27.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-03-20T18:53:27.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-03-20T18:53:28.672 INFO:teuthology.orchestra.run.smithi046.stdout:34359738378 2023-03-20T18:53:30.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-20T18:53:30.815 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-03-20T18:53:30.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-20T18:53:30.816 INFO:teuthology.orchestra.run.smithi046.stdout:51539607561 2023-03-20T18:53:30.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-20T18:53:31.094 INFO:teuthology.orchestra.run.smithi046.stdout:34359738379 2023-03-20T18:53:33.086 INFO:teuthology.orchestra.run.smithi046.stdout:51539607561 2023-03-20T18:53:35.077 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-03-20T18:53:35.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738378 got 34359738379 for osd.0 2023-03-20T18:53:35.078 DEBUG:teuthology.parallel:result is None 2023-03-20T18:53:37.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607561 got 51539607561 for osd.1 2023-03-20T18:53:37.427 DEBUG:teuthology.parallel:result is None 2023-03-20T18:53:37.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-03-20T18:53:37.428 DEBUG:teuthology.parallel:result is None 2023-03-20T18:53:37.428 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-20T18:53:37.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:53:37.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:53:39.356 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-03-20T18:53:37.631837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17144,"kb_used_data":1376,"kb_used_omap":0,"kb_used_meta":15744,"kb_avail":283098376,"statfs":{"total":289910292480,"available":289892737024,"internally_reserved":0,"allocated":1409024,"data_stored":1050729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16121856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003118"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.035135+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:53:13.035135+0000","last_peered":"2023-03-20T18:53:13.035135+0000","last_clean":"2023-03-20T18:53:13.035135+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:53:13.035135+0000","last_undegraded":"2023-03-20T18:53:13.035135+0000","last_fullsized":"2023-03-20T18:53:13.035135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:21:14.194017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.034788+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:53:13.034788+0000","last_peered":"2023-03-20T18:53:13.034788+0000","last_clean":"2023-03-20T18:53:13.034788+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:53:13.034788+0000","last_undegraded":"2023-03-20T18:53:13.034788+0000","last_fullsized":"2023-03-20T18:53:13.034788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:53:17.445676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.034745+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:53:13.034745+0000","last_peered":"2023-03-20T18:53:13.034745+0000","last_clean":"2023-03-20T18:53:13.034745+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:53:13.034745+0000","last_undegraded":"2023-03-20T18:53:13.034745+0000","last_fullsized":"2023-03-20T18:53:13.034745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:13:01.214700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031469+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:53:13.031469+0000","last_peered":"2023-03-20T18:53:13.031469+0000","last_clean":"2023-03-20T18:53:13.031469+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:53:13.031469+0000","last_undegraded":"2023-03-20T18:53:13.031469+0000","last_fullsized":"2023-03-20T18:53:13.031469+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:17:09.655195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.033364+0000","last_change":"2023-03-20T18:53:13.031159+0000","last_active":"2023-03-20T18:53:13.033364+0000","last_peered":"2023-03-20T18:53:13.033364+0000","last_clean":"2023-03-20T18:53:13.033364+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:53:13.033364+0000","last_undegraded":"2023-03-20T18:53:13.033364+0000","last_fullsized":"2023-03-20T18:53:13.033364+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:43:42.389315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031720+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:53:13.031720+0000","last_peered":"2023-03-20T18:53:13.031720+0000","last_clean":"2023-03-20T18:53:13.031720+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:53:13.031720+0000","last_undegraded":"2023-03-20T18:53:13.031720+0000","last_fullsized":"2023-03-20T18:53:13.031720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:17:35.273446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031726+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:53:13.031726+0000","last_peered":"2023-03-20T18:53:13.031726+0000","last_clean":"2023-03-20T18:53:13.031726+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:53:13.031726+0000","last_undegraded":"2023-03-20T18:53:13.031726+0000","last_fullsized":"2023-03-20T18:53:13.031726+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:29:52.559510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.032160+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:53:13.032160+0000","last_peered":"2023-03-20T18:53:13.032160+0000","last_clean":"2023-03-20T18:53:13.032160+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:53:13.032160+0000","last_undegraded":"2023-03-20T18:53:13.032160+0000","last_fullsized":"2023-03-20T18:53:13.032160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:20:24.574617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'97","reported_seq":139,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.027781+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:53:13.027781+0000","last_peered":"2023-03-20T18:53:13.027781+0000","last_clean":"2023-03-20T18:53:13.027781+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:53:13.027781+0000","last_undegraded":"2023-03-20T18:53:13.027781+0000","last_fullsized":"2023-03-20T18:53:13.027781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:20:39.568795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":38939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607563,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365844,"statfs":{"total":96636764160,"available":96630624256,"internally_reserved":0,"allocated":626688,"data_stored":505895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365844,"statfs":{"total":96636764160,"available":96630624256,"internally_reserved":0,"allocated":626688,"data_stored":505895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:53:39.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:53:39.647 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:53:41.846 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-03-20T18:53:39.632317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17144,"kb_used_data":1376,"kb_used_omap":0,"kb_used_meta":15744,"kb_avail":283098376,"statfs":{"total":289910292480,"available":289892737024,"internally_reserved":0,"allocated":1409024,"data_stored":1050729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16121856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.035135+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:53:13.035135+0000","last_peered":"2023-03-20T18:53:13.035135+0000","last_clean":"2023-03-20T18:53:13.035135+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:53:13.035135+0000","last_undegraded":"2023-03-20T18:53:13.035135+0000","last_fullsized":"2023-03-20T18:53:13.035135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:21:14.194017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.034788+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:53:13.034788+0000","last_peered":"2023-03-20T18:53:13.034788+0000","last_clean":"2023-03-20T18:53:13.034788+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:53:13.034788+0000","last_undegraded":"2023-03-20T18:53:13.034788+0000","last_fullsized":"2023-03-20T18:53:13.034788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:53:17.445676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.034745+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:53:13.034745+0000","last_peered":"2023-03-20T18:53:13.034745+0000","last_clean":"2023-03-20T18:53:13.034745+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:53:13.034745+0000","last_undegraded":"2023-03-20T18:53:13.034745+0000","last_fullsized":"2023-03-20T18:53:13.034745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:13:01.214700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031469+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:53:13.031469+0000","last_peered":"2023-03-20T18:53:13.031469+0000","last_clean":"2023-03-20T18:53:13.031469+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:53:13.031469+0000","last_undegraded":"2023-03-20T18:53:13.031469+0000","last_fullsized":"2023-03-20T18:53:13.031469+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:17:09.655195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.033364+0000","last_change":"2023-03-20T18:53:13.031159+0000","last_active":"2023-03-20T18:53:13.033364+0000","last_peered":"2023-03-20T18:53:13.033364+0000","last_clean":"2023-03-20T18:53:13.033364+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:53:13.033364+0000","last_undegraded":"2023-03-20T18:53:13.033364+0000","last_fullsized":"2023-03-20T18:53:13.033364+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:43:42.389315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031720+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:53:13.031720+0000","last_peered":"2023-03-20T18:53:13.031720+0000","last_clean":"2023-03-20T18:53:13.031720+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:53:13.031720+0000","last_undegraded":"2023-03-20T18:53:13.031720+0000","last_fullsized":"2023-03-20T18:53:13.031720+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:17:35.273446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.031726+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:53:13.031726+0000","last_peered":"2023-03-20T18:53:13.031726+0000","last_clean":"2023-03-20T18:53:13.031726+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:53:13.031726+0000","last_undegraded":"2023-03-20T18:53:13.031726+0000","last_fullsized":"2023-03-20T18:53:13.031726+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:29:52.559510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.032160+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:53:13.032160+0000","last_peered":"2023-03-20T18:53:13.032160+0000","last_clean":"2023-03-20T18:53:13.032160+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:53:13.032160+0000","last_undegraded":"2023-03-20T18:53:13.032160+0000","last_fullsized":"2023-03-20T18:53:13.032160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:20:24.574617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'97","reported_seq":139,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-03-20T18:53:13.027781+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:53:13.027781+0000","last_peered":"2023-03-20T18:53:13.027781+0000","last_clean":"2023-03-20T18:53:13.027781+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:53:13.027781+0000","last_undegraded":"2023-03-20T18:53:13.027781+0000","last_fullsized":"2023-03-20T18:53:13.027781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:20:39.568795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":38939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607563,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365844,"statfs":{"total":96636764160,"available":96630624256,"internally_reserved":0,"allocated":626688,"data_stored":505895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365844,"statfs":{"total":96636764160,"available":96630624256,"internally_reserved":0,"allocated":626688,"data_stored":505895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:53:42.458 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-20T18:53:42.458 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-20T18:53:42.459 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-03-20T18:53:42.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-03-20T18:53:42.776 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:53:44.746 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-20T18:53:44.747 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-03-20T18:53:44.747 INFO:teuthology.run_tasks:Running task workunit... 2023-03-20T18:53:46.973 INFO:tasks.workunit:Pulling workunits from ref a6dd66830c12498d02f3894e352dd8b49a7bab4c 2023-03-20T18:53:50.446 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-03-20T18:53:50.446 INFO:tasks.workunit:timeout=3h 2023-03-20T18:53:50.446 INFO:tasks.workunit:cleanup=True 2023-03-20T18:53:50.447 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2023-03-20T18:53:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:53:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-03-20T18:53:53.055 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-03-20T18:53:53.062 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-03-20T18:53:54.553 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-03-20T18:53:54.561 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a6dd66830c12498d02f3894e352dd8b49a7bab4c 2023-03-20T18:53:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-03-20T18:54:38.144 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to 'a6dd66830c12498d02f3894e352dd8b49a7bab4c'. 2023-03-20T18:54:39.601 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-03-20T18:54:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:54:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at a6dd66830c1 Merge pull request #50565 from zdover23/wip-doc-2023-03-17-backport-50563-to-reef 2023-03-20T18:54:40.625 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-03-20T18:54:40.691 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-03-20T18:54:41.526 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-20T18:54:41.526 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-03-20T18:54:41.565 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-03-20T18:54:42.510 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-03-20T18:54:42.510 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-20T18:54:42.510 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-20T18:54:42.510 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-03-20T18:54:42.510 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-20T18:54:42.511 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-03-20T18:54:44.022 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-03-20T18:54:44.031 INFO:tasks.workunit:Running workunits matching cephtool on client.0... 2023-03-20T18:54:45.857 INFO:tasks.workunit:Running workunit cephtool/test.sh... 2023-03-20T18:54:45.857 DEBUG:teuthology.orchestra.run.smithi046:workunit test cephtool/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2023-03-20T18:54:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2023-03-20T18:54:47.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh 2023-03-20T18:54:47.153 INFO:tasks.workunit.client.0.smithi046.stderr:++ TIMEOUT=300 2023-03-20T18:54:47.153 INFO:tasks.workunit.client.0.smithi046.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2023-03-20T18:54:48.538 INFO:tasks.workunit.client.0.smithi046.stderr:++ MAX_TIMEOUT=15 2023-03-20T18:54:48.538 INFO:tasks.workunit.client.0.smithi046.stderr:++ PG_NUM=4 2023-03-20T18:54:48.539 INFO:tasks.workunit.client.0.smithi046.stderr:++ TMPDIR=/tmp 2023-03-20T18:54:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2023-03-20T18:54:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ TESTDIR=/home/ubuntu/cephtest 2023-03-20T18:54:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:++ type xmlstarlet 2023-03-20T18:54:53.342 INFO:tasks.workunit.client.0.smithi046.stderr:++ XMLSTARLET=xmlstarlet 2023-03-20T18:54:53.342 INFO:tasks.workunit.client.0.smithi046.stderr:+++ uname 2023-03-20T18:54:53.343 INFO:tasks.workunit.client.0.smithi046.stderr:++ '[' Linux = FreeBSD ']' 2023-03-20T18:54:53.343 INFO:tasks.workunit.client.0.smithi046.stderr:++ SED=sed 2023-03-20T18:54:53.343 INFO:tasks.workunit.client.0.smithi046.stderr:++ AWK=awk 2023-03-20T18:54:54.547 INFO:tasks.workunit.client.0.smithi046.stderr:+++ stty -a 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:+++ head -1 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:++ termwidth= 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:++ DIFFCOLOPTS='-y ' 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:++ KERNCORE=kernel.core_pattern 2023-03-20T18:54:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:++ EXTRA_OPTS= 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:++ test '' = TESTS 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -o functrace 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:11: : SUDO=sudo 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : export CEPH_DEV=1 2023-03-20T18:54:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : CEPH_DEV=1 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : mktemp -d /tmp/cephtool.XXX 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : TEMP_DIR=/tmp/cephtool.yGC 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:46: : trap 'rm -fr /tmp/cephtool.yGC' 0 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : mktemp /tmp/cephtool.yGC/test_invalid.XXX 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : TMPFILE=/tmp/cephtool.yGC/test_invalid.vQy 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2849: : set +x 2023-03-20T18:54:54.550 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' already exists 2023-03-20T18:54:55.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2981: : test_mon_injectargs 2023-03-20T18:54:55.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:201: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2023-03-20T18:54:55.903 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:54:55.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-03-20T18:54:55.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: grep false 2023-03-20T18:54:55.905 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_enable_op_tracker": "false" 2023-03-20T18:54:55.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:203: test_mon_injectargs: ceph tell osd.0 injectargs '--osd_enable_op_tracker --osd_op_history_duration 500' 2023-03-20T18:54:55.906 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:54:55.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-03-20T18:54:55.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: grep true 2023-03-20T18:54:55.907 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_enable_op_tracker": "true" 2023-03-20T18:54:55.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2023-03-20T18:54:56.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: grep 500 2023-03-20T18:54:56.733 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_op_history_duration": "500" 2023-03-20T18:54:56.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:206: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2023-03-20T18:54:56.734 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:54:58.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-03-20T18:54:58.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: grep false 2023-03-20T18:54:58.650 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_enable_op_tracker": "false" 2023-03-20T18:55:02.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:208: test_mon_injectargs: ceph tell osd.0 injectargs -- --osd_enable_op_tracker 2023-03-20T18:55:02.945 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:02.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-03-20T18:55:02.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: grep true 2023-03-20T18:55:02.946 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_enable_op_tracker": "true" 2023-03-20T18:55:02.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:210: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_enable_op_tracker --osd_op_history_duration 600' 2023-03-20T18:55:02.947 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:02.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-03-20T18:55:02.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: grep true 2023-03-20T18:55:02.948 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_enable_op_tracker": "true" 2023-03-20T18:55:02.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2023-03-20T18:55:05.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: grep 600 2023-03-20T18:55:05.824 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_op_history_duration": "600" 2023-03-20T18:55:05.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:214: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_deep_scrub_interval 2419200' 2023-03-20T18:55:05.824 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: ceph tell osd.0 config get osd_deep_scrub_interval 2023-03-20T18:55:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: grep 2419200 2023-03-20T18:55:05.825 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_deep_scrub_interval": "2419200.000000" 2023-03-20T18:55:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:217: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon_probe_timeout 2' 2023-03-20T18:55:05.825 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: ceph tell osd.0 config get mon_probe_timeout 2023-03-20T18:55:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: grep 2 2023-03-20T18:55:05.826 INFO:tasks.workunit.client.0.smithi046.stdout: "mon_probe_timeout": "2.000000" 2023-03-20T18:55:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:220: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon-lease 6' 2023-03-20T18:55:08.068 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:08.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: grep 6 2023-03-20T18:55:08.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: ceph tell osd.0 config get mon_lease 2023-03-20T18:55:08.068 INFO:tasks.workunit.client.0.smithi046.stdout: "mon_lease": "6.000000" 2023-03-20T18:55:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:224: test_mon_injectargs: expect_false ceph tell osd.0 injectargs --osd-scrub-auto-repair-num-errors -1 2023-03-20T18:55:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:225: test_mon_injectargs: check_response 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2023-03-20T18:55:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2023-03-20T18:55:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-03-20T18:55:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-03-20T18:55:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-03-20T18:55:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' /tmp/cephtool.yGC/test_invalid.vQy 2023-03-20T18:55:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:227: test_mon_injectargs: expect_failure /tmp/cephtool.yGC 'Option --osd_op_history_duration requires an argument' ceph tell osd.0 injectargs -- --osd_op_history_duration 2023-03-20T18:55:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1892: expect_failure: local dir=/tmp/cephtool.yGC 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1893: expect_failure: shift 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1894: expect_failure: local 'expected=Option --osd_op_history_duration requires an argument' 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1896: expect_failure: local success 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1898: expect_failure: ceph tell osd.0 injectargs -- --osd_op_history_duration 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1901: expect_failure: success=false 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1904: expect_failure: false 2023-03-20T18:55:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1904: expect_failure: grep --quiet 'Option --osd_op_history_duration requires an argument' /tmp/cephtool.yGC/out 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1908: expect_failure: return 0 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : set +x 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2981: : test_mon_injectargs_SI 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "10000" 2023-03-20T18:55:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-03-20T18:55:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:09.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: initial_value=10000 2023-03-20T18:55:09.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:242: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10 2023-03-20T18:55:09.707 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2023-03-20T18:55:11.396 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "mon_pg_warn_min_objects = '10' (not observed, change may require restart) " 2023-03-20T18:55:11.396 INFO:tasks.workunit.client.0.smithi046.stdout:} 2023-03-20T18:55:11.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:243: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2023-03-20T18:55:11.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:11.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:12.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:12.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2023-03-20T18:55:12.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:12.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:12.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "10" 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:13.271 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2023-03-20T18:55:13.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:13.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2023-03-20T18:55:14.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2023-03-20T18:55:14.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:14.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2023-03-20T18:55:14.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2023-03-20T18:55:14.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:15.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2023-03-20T18:55:15.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2023-03-20T18:55:15.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:244: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10K 2023-03-20T18:55:17.409 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2023-03-20T18:55:19.141 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "mon_pg_warn_min_objects = '10000' (not observed, change may require restart) " 2023-03-20T18:55:19.141 INFO:tasks.workunit.client.0.smithi046.stdout:} 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:245: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:19.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "10000" 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2023-03-20T18:55:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2023-03-20T18:55:19.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:246: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 1G 2023-03-20T18:55:19.145 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2023-03-20T18:55:19.145 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "mon_pg_warn_min_objects = '1000000000' (not observed, change may require restart) " 2023-03-20T18:55:19.145 INFO:tasks.workunit.client.0.smithi046.stdout:} 2023-03-20T18:55:19.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:247: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "1000000000" 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:21.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2023-03-20T18:55:21.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:21.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2023-03-20T18:55:21.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2023-03-20T18:55:21.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:21.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2023-03-20T18:55:21.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:248: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10F 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:249: test_mon_injectargs_SI: check_response '(22) Invalid argument' 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='(22) Invalid argument' 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- '(22) Invalid argument' /tmp/cephtool.yGC/test_invalid.vQy 2023-03-20T18:55:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:251: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10' 2023-03-20T18:55:21.606 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:21.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:252: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:27.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "10" 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2023-03-20T18:55:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:27.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2023-03-20T18:55:30.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2023-03-20T18:55:30.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:253: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10K' 2023-03-20T18:55:30.107 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:254: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:31.777 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "10000" 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2023-03-20T18:55:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2023-03-20T18:55:35.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:255: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 1G' 2023-03-20T18:55:35.394 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:35.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:256: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-03-20T18:55:37.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-03-20T18:55:37.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr: "mon_pg_warn_min_objects": "1000000000" 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:}' 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-03-20T18:55:38.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2023-03-20T18:55:40.682 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2023-03-20T18:55:40.682 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-03-20T18:55:40.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:257: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: Parse error setting mon_pg_warn_min_objects to '10F' using injectargs (strict_si_cast: unit prefix not recognized). 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:55:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:258: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2023-03-20T18:55:43.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:55:43.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2023-03-20T18:55:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: Parse error setting mon_globalid_prealloc to '-1' using injectargs (strict_sistrtoll: value should not be negative). 2023-03-20T18:55:44.723 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-03-20T18:55:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:55:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:259: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10000 2023-03-20T18:55:44.724 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2023-03-20T18:55:44.724 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "mon_pg_warn_min_objects = '10000' (not observed, change may require restart) " 2023-03-20T18:55:44.724 INFO:tasks.workunit.client.0.smithi046.stdout:} 2023-03-20T18:55:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : set +x 2023-03-20T18:55:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2981: : test_tiering_1 2023-03-20T18:55:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:339: test_tiering_1: ceph osd pool create slow 2 2023-03-20T18:55:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'slow' already exists 2023-03-20T18:55:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:340: test_tiering_1: ceph osd pool application enable slow rados 2023-03-20T18:55:44.727 INFO:tasks.workunit.client.0.smithi046.stderr:enabled application 'rados' on pool 'slow' 2023-03-20T18:55:44.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:341: test_tiering_1: ceph osd pool create slow2 2 2023-03-20T18:55:44.727 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'slow2' already exists 2023-03-20T18:55:44.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:342: test_tiering_1: ceph osd pool application enable slow2 rados 2023-03-20T18:55:48.193 INFO:tasks.workunit.client.0.smithi046.stderr:enabled application 'rados' on pool 'slow2' 2023-03-20T18:55:48.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:343: test_tiering_1: ceph osd pool create cache 2 2023-03-20T18:55:49.438 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'cache' already exists 2023-03-20T18:55:49.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:344: test_tiering_1: ceph osd pool create cache2 2 2023-03-20T18:55:49.439 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'cache2' already exists 2023-03-20T18:55:50.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:345: test_tiering_1: ceph osd tier add slow cache 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'cache' is now (or already was) a tier of 'slow' 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:346: test_tiering_1: ceph osd tier add slow cache2 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'cache2' is now (or already was) a tier of 'slow' 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:347: test_tiering_1: expect_false ceph osd tier add slow2 cache 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add slow2 cache 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: tier pool 'cache' is already a tier of 'slow' 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: ceph osd pool ls detail -f json 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: jq '.[] | select(.pool_name == "slow") | .application_metadata["rados"]' 2023-03-20T18:55:50.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: grep '{}' 2023-03-20T18:55:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:53.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: grep '{}' 2023-03-20T18:55:53.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: ceph osd pool ls detail -f json 2023-03-20T18:55:53.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: jq '.[] | select(.pool_name == "slow2") | .application_metadata["rados"]' 2023-03-20T18:55:53.574 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:53.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: ceph osd pool ls detail -f json 2023-03-20T18:55:54.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: grep '{}' 2023-03-20T18:55:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: jq '.[] | select(.pool_name == "cache") | .application_metadata["rados"]' 2023-03-20T18:55:55.862 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:55.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: ceph osd pool ls detail -f json 2023-03-20T18:55:55.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: jq '.[] | select(.pool_name == "cache2") | .application_metadata["rados"]' 2023-03-20T18:55:55.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: grep '{}' 2023-03-20T18:55:58.264 INFO:tasks.workunit.client.0.smithi046.stdout:{} 2023-03-20T18:55:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-03-20T18:56:02.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:354: test_tiering_1: expect_false ceph osd tier cache-mode cache forward 2023-03-20T18:56:04.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:56:04.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward 2023-03-20T18:56:04.208 INFO:tasks.workunit.client.0.smithi046.stderr:Invalid command: forward not in writeback|readproxy|readonly|none 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: invalid command 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:355: test_tiering_1: expect_false ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:Invalid command: forward not in writeback|readproxy|readonly|none 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: invalid command 2023-03-20T18:56:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:357: test_tiering_1: ceph osd tier cache-mode cache writeback 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:set cache-mode for pool 'cache' to writeback 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:358: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly 2023-03-20T18:56:06.013 INFO:tasks.workunit.client.0.smithi046.stderr:Error EPERM: 'readonly' is not a well-supported cache mode and may corrupt your data. pass --yes-i-really-mean-it to force. 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:359: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: unable to set cache-mode 'readonly' on a 'writeback' pool; only 'readproxy' allowed. 2023-03-20T18:56:06.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-03-20T18:56:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:360: test_tiering_1: ceph osd tier cache-mode cache proxy 2023-03-20T18:56:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:Invalid command: proxy not in writeback|readproxy|readonly|none 2023-03-20T18:56:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2023-03-20T18:56:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: invalid command 2023-03-20T18:56:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_tiering_1: rm -fr /tmp/cephtool.yGC 2023-03-20T18:56:07.058 INFO:tasks.workunit:Stopping ['cephtool'] on client.0... 2023-03-20T18:56:07.058 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-03-20T18:56:07.316 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi046 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2023-03-20T18:56:08.532 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=633652ad6fdb406c890cd795dd1937ca Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a6dd66830c12498d02f3894e352dd8b49a7bab4c/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi046 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2023-03-20T18:56:09.831 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-03-20T18:56:09.842 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-20T18:56:10.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:56:11.095 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:56:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:56:13.785 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2023-03-20T18:56:09.672685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17792,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":283097728,"statfs":{"total":289910292480,"available":289892073472,"internally_reserved":0,"allocated":1679360,"data_stored":1184778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"1.0","version":"14'97","reported_seq":150,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942726+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:55:05.942726+0000","last_peered":"2023-03-20T18:55:05.942726+0000","last_clean":"2023-03-20T18:55:05.942726+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:55:05.942726+0000","last_undegraded":"2023-03-20T18:55:05.942726+0000","last_fullsized":"2023-03-20T18:55:05.942726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:20:39.568795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941991+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:55:05.941991+0000","last_peered":"2023-03-20T18:55:05.941991+0000","last_clean":"2023-03-20T18:55:05.941991+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:55:05.941991+0000","last_undegraded":"2023-03-20T18:55:05.941991+0000","last_fullsized":"2023-03-20T18:55:05.941991+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:20:24.574617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364547+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:55:03.364547+0000","last_peered":"2023-03-20T18:55:03.364547+0000","last_clean":"2023-03-20T18:55:03.364547+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:55:03.364547+0000","last_undegraded":"2023-03-20T18:55:03.364547+0000","last_fullsized":"2023-03-20T18:55:03.364547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:29:52.559510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365432+0000","last_change":"2023-03-20T18:54:55.287821+0000","last_active":"2023-03-20T18:55:03.365432+0000","last_peered":"2023-03-20T18:55:03.365432+0000","last_clean":"2023-03-20T18:55:03.365432+0000","last_became_active":"2023-03-20T18:54:55.287358+0000","last_became_peered":"2023-03-20T18:54:55.287358+0000","last_unstale":"2023-03-20T18:55:03.365432+0000","last_undegraded":"2023-03-20T18:55:03.365432+0000","last_fullsized":"2023-03-20T18:55:03.365432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T05:55:54.423712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364927+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:55:03.364927+0000","last_peered":"2023-03-20T18:55:03.364927+0000","last_clean":"2023-03-20T18:55:03.364927+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:55:03.364927+0000","last_undegraded":"2023-03-20T18:55:03.364927+0000","last_fullsized":"2023-03-20T18:55:03.364927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:17:35.273446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942577+0000","last_change":"2023-03-20T18:54:55.286337+0000","last_active":"2023-03-20T18:55:05.942577+0000","last_peered":"2023-03-20T18:55:05.942577+0000","last_clean":"2023-03-20T18:55:05.942577+0000","last_became_active":"2023-03-20T18:54:55.285843+0000","last_became_peered":"2023-03-20T18:54:55.285843+0000","last_unstale":"2023-03-20T18:55:05.942577+0000","last_undegraded":"2023-03-20T18:55:05.942577+0000","last_fullsized":"2023-03-20T18:55:05.942577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:54:17.320369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365424+0000","last_change":"2023-03-20T18:53:13.031159+0000","last_active":"2023-03-20T18:55:03.365424+0000","last_peered":"2023-03-20T18:55:03.365424+0000","last_clean":"2023-03-20T18:55:03.365424+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:55:03.365424+0000","last_undegraded":"2023-03-20T18:55:03.365424+0000","last_fullsized":"2023-03-20T18:55:03.365424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:43:42.389315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.363649+0000","last_change":"2023-03-20T18:55:02.367518+0000","last_active":"2023-03-20T18:55:03.363649+0000","last_peered":"2023-03-20T18:55:03.363649+0000","last_clean":"2023-03-20T18:55:03.363649+0000","last_became_active":"2023-03-20T18:55:02.367219+0000","last_became_peered":"2023-03-20T18:55:02.367219+0000","last_unstale":"2023-03-20T18:55:03.363649+0000","last_undegraded":"2023-03-20T18:55:03.363649+0000","last_fullsized":"2023-03-20T18:55:03.363649+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T20:30:48.923860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941999+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:55:05.941999+0000","last_peered":"2023-03-20T18:55:05.941999+0000","last_clean":"2023-03-20T18:55:05.941999+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:55:05.941999+0000","last_undegraded":"2023-03-20T18:55:05.941999+0000","last_fullsized":"2023-03-20T18:55:05.941999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:17:09.655195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942625+0000","last_change":"2023-03-20T18:55:02.364036+0000","last_active":"2023-03-20T18:55:05.942625+0000","last_peered":"2023-03-20T18:55:05.942625+0000","last_clean":"2023-03-20T18:55:05.942625+0000","last_became_active":"2023-03-20T18:55:02.363649+0000","last_became_peered":"2023-03-20T18:55:02.363649+0000","last_unstale":"2023-03-20T18:55:05.942625+0000","last_undegraded":"2023-03-20T18:55:05.942625+0000","last_fullsized":"2023-03-20T18:55:05.942625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:27:32.149815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941961+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:55:05.941961+0000","last_peered":"2023-03-20T18:55:05.941961+0000","last_clean":"2023-03-20T18:55:05.941961+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:55:05.941961+0000","last_undegraded":"2023-03-20T18:55:05.941961+0000","last_fullsized":"2023-03-20T18:55:05.941961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:13:01.214700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942836+0000","last_change":"2023-03-20T18:55:01.357464+0000","last_active":"2023-03-20T18:55:05.942836+0000","last_peered":"2023-03-20T18:55:05.942836+0000","last_clean":"2023-03-20T18:55:05.942836+0000","last_became_active":"2023-03-20T18:55:01.356978+0000","last_became_peered":"2023-03-20T18:55:01.356978+0000","last_unstale":"2023-03-20T18:55:05.942836+0000","last_undegraded":"2023-03-20T18:55:05.942836+0000","last_fullsized":"2023-03-20T18:55:05.942836+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:12:03.450754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942121+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:55:05.942121+0000","last_peered":"2023-03-20T18:55:05.942121+0000","last_clean":"2023-03-20T18:55:05.942121+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:55:05.942121+0000","last_undegraded":"2023-03-20T18:55:05.942121+0000","last_fullsized":"2023-03-20T18:55:05.942121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:53:17.445676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364026+0000","last_change":"2023-03-20T18:54:59.003766+0000","last_active":"2023-03-20T18:55:03.364026+0000","last_peered":"2023-03-20T18:55:03.364026+0000","last_clean":"2023-03-20T18:55:03.364026+0000","last_became_active":"2023-03-20T18:54:59.002918+0000","last_became_peered":"2023-03-20T18:54:59.002918+0000","last_unstale":"2023-03-20T18:55:03.364026+0000","last_undegraded":"2023-03-20T18:55:03.364026+0000","last_fullsized":"2023-03-20T18:55:03.364026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T01:10:01.900871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364333+0000","last_change":"2023-03-20T18:55:01.356228+0000","last_active":"2023-03-20T18:55:03.364333+0000","last_peered":"2023-03-20T18:55:03.364333+0000","last_clean":"2023-03-20T18:55:03.364333+0000","last_became_active":"2023-03-20T18:55:01.355775+0000","last_became_peered":"2023-03-20T18:55:01.355775+0000","last_unstale":"2023-03-20T18:55:03.364333+0000","last_undegraded":"2023-03-20T18:55:03.364333+0000","last_fullsized":"2023-03-20T18:55:03.364333+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:48:50.272984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942222+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:55:05.942222+0000","last_peered":"2023-03-20T18:55:05.942222+0000","last_clean":"2023-03-20T18:55:05.942222+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:55:05.942222+0000","last_undegraded":"2023-03-20T18:55:05.942222+0000","last_fullsized":"2023-03-20T18:55:05.942222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:21:14.194017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364561+0000","last_change":"2023-03-20T18:54:58.323785+0000","last_active":"2023-03-20T18:55:03.364561+0000","last_peered":"2023-03-20T18:55:03.364561+0000","last_clean":"2023-03-20T18:55:03.364561+0000","last_became_active":"2023-03-20T18:54:58.323411+0000","last_became_peered":"2023-03-20T18:54:58.323411+0000","last_unstale":"2023-03-20T18:55:03.364561+0000","last_undegraded":"2023-03-20T18:55:03.364561+0000","last_fullsized":"2023-03-20T18:55:03.364561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:32:43.388503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":5,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":4,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":3,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5368,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94366472,"statfs":{"total":96636764160,"available":96631267328,"internally_reserved":0,"allocated":245760,"data_stored":83622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.39800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.46200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.51100000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607593,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:55:02 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.53100000000000003,"15min":0.54500000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.67300000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Mar 20 18:55:07 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.59599999999999997,"15min":0.61199999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66000000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738411,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.45800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61499999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:56:14.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:56:14.592 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:56:15.861 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:56:15.861 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2023-03-20T18:56:13.673851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17792,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":283097728,"statfs":{"total":289910292480,"available":289892073472,"internally_reserved":0,"allocated":1679360,"data_stored":1184778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003375"},"pg_stats":[{"pgid":"1.0","version":"14'97","reported_seq":150,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942726+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:55:05.942726+0000","last_peered":"2023-03-20T18:55:05.942726+0000","last_clean":"2023-03-20T18:55:05.942726+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:55:05.942726+0000","last_undegraded":"2023-03-20T18:55:05.942726+0000","last_fullsized":"2023-03-20T18:55:05.942726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:20:39.568795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941991+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:55:05.941991+0000","last_peered":"2023-03-20T18:55:05.941991+0000","last_clean":"2023-03-20T18:55:05.941991+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:55:05.941991+0000","last_undegraded":"2023-03-20T18:55:05.941991+0000","last_fullsized":"2023-03-20T18:55:05.941991+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:20:24.574617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364547+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:55:03.364547+0000","last_peered":"2023-03-20T18:55:03.364547+0000","last_clean":"2023-03-20T18:55:03.364547+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:55:03.364547+0000","last_undegraded":"2023-03-20T18:55:03.364547+0000","last_fullsized":"2023-03-20T18:55:03.364547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:29:52.559510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365432+0000","last_change":"2023-03-20T18:54:55.287821+0000","last_active":"2023-03-20T18:55:03.365432+0000","last_peered":"2023-03-20T18:55:03.365432+0000","last_clean":"2023-03-20T18:55:03.365432+0000","last_became_active":"2023-03-20T18:54:55.287358+0000","last_became_peered":"2023-03-20T18:54:55.287358+0000","last_unstale":"2023-03-20T18:55:03.365432+0000","last_undegraded":"2023-03-20T18:55:03.365432+0000","last_fullsized":"2023-03-20T18:55:03.365432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T05:55:54.423712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364927+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:55:03.364927+0000","last_peered":"2023-03-20T18:55:03.364927+0000","last_clean":"2023-03-20T18:55:03.364927+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:55:03.364927+0000","last_undegraded":"2023-03-20T18:55:03.364927+0000","last_fullsized":"2023-03-20T18:55:03.364927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:17:35.273446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942577+0000","last_change":"2023-03-20T18:54:55.286337+0000","last_active":"2023-03-20T18:55:05.942577+0000","last_peered":"2023-03-20T18:55:05.942577+0000","last_clean":"2023-03-20T18:55:05.942577+0000","last_became_active":"2023-03-20T18:54:55.285843+0000","last_became_peered":"2023-03-20T18:54:55.285843+0000","last_unstale":"2023-03-20T18:55:05.942577+0000","last_undegraded":"2023-03-20T18:55:05.942577+0000","last_fullsized":"2023-03-20T18:55:05.942577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:54:17.320369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365424+0000","last_change":"2023-03-20T18:53:13.031159+0000","last_active":"2023-03-20T18:55:03.365424+0000","last_peered":"2023-03-20T18:55:03.365424+0000","last_clean":"2023-03-20T18:55:03.365424+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:55:03.365424+0000","last_undegraded":"2023-03-20T18:55:03.365424+0000","last_fullsized":"2023-03-20T18:55:03.365424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:43:42.389315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.363649+0000","last_change":"2023-03-20T18:55:02.367518+0000","last_active":"2023-03-20T18:55:03.363649+0000","last_peered":"2023-03-20T18:55:03.363649+0000","last_clean":"2023-03-20T18:55:03.363649+0000","last_became_active":"2023-03-20T18:55:02.367219+0000","last_became_peered":"2023-03-20T18:55:02.367219+0000","last_unstale":"2023-03-20T18:55:03.363649+0000","last_undegraded":"2023-03-20T18:55:03.363649+0000","last_fullsized":"2023-03-20T18:55:03.363649+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T20:30:48.923860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941999+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:55:05.941999+0000","last_peered":"2023-03-20T18:55:05.941999+0000","last_clean":"2023-03-20T18:55:05.941999+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:55:05.941999+0000","last_undegraded":"2023-03-20T18:55:05.941999+0000","last_fullsized":"2023-03-20T18:55:05.941999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:17:09.655195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942625+0000","last_change":"2023-03-20T18:55:02.364036+0000","last_active":"2023-03-20T18:55:05.942625+0000","last_peered":"2023-03-20T18:55:05.942625+0000","last_clean":"2023-03-20T18:55:05.942625+0000","last_became_active":"2023-03-20T18:55:02.363649+0000","last_became_peered":"2023-03-20T18:55:02.363649+0000","last_unstale":"2023-03-20T18:55:05.942625+0000","last_undegraded":"2023-03-20T18:55:05.942625+0000","last_fullsized":"2023-03-20T18:55:05.942625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:27:32.149815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941961+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:55:05.941961+0000","last_peered":"2023-03-20T18:55:05.941961+0000","last_clean":"2023-03-20T18:55:05.941961+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:55:05.941961+0000","last_undegraded":"2023-03-20T18:55:05.941961+0000","last_fullsized":"2023-03-20T18:55:05.941961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:13:01.214700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942836+0000","last_change":"2023-03-20T18:55:01.357464+0000","last_active":"2023-03-20T18:55:05.942836+0000","last_peered":"2023-03-20T18:55:05.942836+0000","last_clean":"2023-03-20T18:55:05.942836+0000","last_became_active":"2023-03-20T18:55:01.356978+0000","last_became_peered":"2023-03-20T18:55:01.356978+0000","last_unstale":"2023-03-20T18:55:05.942836+0000","last_undegraded":"2023-03-20T18:55:05.942836+0000","last_fullsized":"2023-03-20T18:55:05.942836+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:12:03.450754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942121+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:55:05.942121+0000","last_peered":"2023-03-20T18:55:05.942121+0000","last_clean":"2023-03-20T18:55:05.942121+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:55:05.942121+0000","last_undegraded":"2023-03-20T18:55:05.942121+0000","last_fullsized":"2023-03-20T18:55:05.942121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:53:17.445676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364026+0000","last_change":"2023-03-20T18:54:59.003766+0000","last_active":"2023-03-20T18:55:03.364026+0000","last_peered":"2023-03-20T18:55:03.364026+0000","last_clean":"2023-03-20T18:55:03.364026+0000","last_became_active":"2023-03-20T18:54:59.002918+0000","last_became_peered":"2023-03-20T18:54:59.002918+0000","last_unstale":"2023-03-20T18:55:03.364026+0000","last_undegraded":"2023-03-20T18:55:03.364026+0000","last_fullsized":"2023-03-20T18:55:03.364026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T01:10:01.900871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364333+0000","last_change":"2023-03-20T18:55:01.356228+0000","last_active":"2023-03-20T18:55:03.364333+0000","last_peered":"2023-03-20T18:55:03.364333+0000","last_clean":"2023-03-20T18:55:03.364333+0000","last_became_active":"2023-03-20T18:55:01.355775+0000","last_became_peered":"2023-03-20T18:55:01.355775+0000","last_unstale":"2023-03-20T18:55:03.364333+0000","last_undegraded":"2023-03-20T18:55:03.364333+0000","last_fullsized":"2023-03-20T18:55:03.364333+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:48:50.272984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942222+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:55:05.942222+0000","last_peered":"2023-03-20T18:55:05.942222+0000","last_clean":"2023-03-20T18:55:05.942222+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:55:05.942222+0000","last_undegraded":"2023-03-20T18:55:05.942222+0000","last_fullsized":"2023-03-20T18:55:05.942222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:21:14.194017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364561+0000","last_change":"2023-03-20T18:54:58.323785+0000","last_active":"2023-03-20T18:55:03.364561+0000","last_peered":"2023-03-20T18:55:03.364561+0000","last_clean":"2023-03-20T18:55:03.364561+0000","last_became_active":"2023-03-20T18:54:58.323411+0000","last_became_peered":"2023-03-20T18:54:58.323411+0000","last_unstale":"2023-03-20T18:55:03.364561+0000","last_undegraded":"2023-03-20T18:55:03.364561+0000","last_fullsized":"2023-03-20T18:55:03.364561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:32:43.388503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":5,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":4,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":3,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5368,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94366472,"statfs":{"total":96636764160,"available":96631267328,"internally_reserved":0,"allocated":245760,"data_stored":83622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.39800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.46200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.621}]}]},{"osd":1,"up_from":12,"seq":51539607594,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.625,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55000000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Mar 20 18:56:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61599999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.48499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738411,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.45800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61499999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:56:16.474 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-20T18:56:17.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:56:18.280 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:56:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:56:19.067 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2023-03-20T18:56:17.674944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17792,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":283097728,"statfs":{"total":289910292480,"available":289892073472,"internally_reserved":0,"allocated":1679360,"data_stored":1184778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003365"},"pg_stats":[{"pgid":"1.0","version":"14'97","reported_seq":150,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942726+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:55:05.942726+0000","last_peered":"2023-03-20T18:55:05.942726+0000","last_clean":"2023-03-20T18:55:05.942726+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:55:05.942726+0000","last_undegraded":"2023-03-20T18:55:05.942726+0000","last_fullsized":"2023-03-20T18:55:05.942726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:20:39.568795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941991+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:55:05.941991+0000","last_peered":"2023-03-20T18:55:05.941991+0000","last_clean":"2023-03-20T18:55:05.941991+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:55:05.941991+0000","last_undegraded":"2023-03-20T18:55:05.941991+0000","last_fullsized":"2023-03-20T18:55:05.941991+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:20:24.574617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364547+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:55:03.364547+0000","last_peered":"2023-03-20T18:55:03.364547+0000","last_clean":"2023-03-20T18:55:03.364547+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:55:03.364547+0000","last_undegraded":"2023-03-20T18:55:03.364547+0000","last_fullsized":"2023-03-20T18:55:03.364547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:29:52.559510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365432+0000","last_change":"2023-03-20T18:54:55.287821+0000","last_active":"2023-03-20T18:55:03.365432+0000","last_peered":"2023-03-20T18:55:03.365432+0000","last_clean":"2023-03-20T18:55:03.365432+0000","last_became_active":"2023-03-20T18:54:55.287358+0000","last_became_peered":"2023-03-20T18:54:55.287358+0000","last_unstale":"2023-03-20T18:55:03.365432+0000","last_undegraded":"2023-03-20T18:55:03.365432+0000","last_fullsized":"2023-03-20T18:55:03.365432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T05:55:54.423712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364927+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:55:03.364927+0000","last_peered":"2023-03-20T18:55:03.364927+0000","last_clean":"2023-03-20T18:55:03.364927+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:55:03.364927+0000","last_undegraded":"2023-03-20T18:55:03.364927+0000","last_fullsized":"2023-03-20T18:55:03.364927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:17:35.273446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942577+0000","last_change":"2023-03-20T18:54:55.286337+0000","last_active":"2023-03-20T18:55:05.942577+0000","last_peered":"2023-03-20T18:55:05.942577+0000","last_clean":"2023-03-20T18:55:05.942577+0000","last_became_active":"2023-03-20T18:54:55.285843+0000","last_became_peered":"2023-03-20T18:54:55.285843+0000","last_unstale":"2023-03-20T18:55:05.942577+0000","last_undegraded":"2023-03-20T18:55:05.942577+0000","last_fullsized":"2023-03-20T18:55:05.942577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:54:17.320369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.365424+0000","last_change":"2023-03-20T18:53:13.031159+0000","last_active":"2023-03-20T18:55:03.365424+0000","last_peered":"2023-03-20T18:55:03.365424+0000","last_clean":"2023-03-20T18:55:03.365424+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:55:03.365424+0000","last_undegraded":"2023-03-20T18:55:03.365424+0000","last_fullsized":"2023-03-20T18:55:03.365424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:43:42.389315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.363649+0000","last_change":"2023-03-20T18:55:02.367518+0000","last_active":"2023-03-20T18:55:03.363649+0000","last_peered":"2023-03-20T18:55:03.363649+0000","last_clean":"2023-03-20T18:55:03.363649+0000","last_became_active":"2023-03-20T18:55:02.367219+0000","last_became_peered":"2023-03-20T18:55:02.367219+0000","last_unstale":"2023-03-20T18:55:03.363649+0000","last_undegraded":"2023-03-20T18:55:03.363649+0000","last_fullsized":"2023-03-20T18:55:03.363649+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T20:30:48.923860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941999+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:55:05.941999+0000","last_peered":"2023-03-20T18:55:05.941999+0000","last_clean":"2023-03-20T18:55:05.941999+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:55:05.941999+0000","last_undegraded":"2023-03-20T18:55:05.941999+0000","last_fullsized":"2023-03-20T18:55:05.941999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:17:09.655195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942625+0000","last_change":"2023-03-20T18:55:02.364036+0000","last_active":"2023-03-20T18:55:05.942625+0000","last_peered":"2023-03-20T18:55:05.942625+0000","last_clean":"2023-03-20T18:55:05.942625+0000","last_became_active":"2023-03-20T18:55:02.363649+0000","last_became_peered":"2023-03-20T18:55:02.363649+0000","last_unstale":"2023-03-20T18:55:05.942625+0000","last_undegraded":"2023-03-20T18:55:05.942625+0000","last_fullsized":"2023-03-20T18:55:05.942625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:27:32.149815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.941961+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:55:05.941961+0000","last_peered":"2023-03-20T18:55:05.941961+0000","last_clean":"2023-03-20T18:55:05.941961+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:55:05.941961+0000","last_undegraded":"2023-03-20T18:55:05.941961+0000","last_fullsized":"2023-03-20T18:55:05.941961+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T06:13:01.214700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942836+0000","last_change":"2023-03-20T18:55:01.357464+0000","last_active":"2023-03-20T18:55:05.942836+0000","last_peered":"2023-03-20T18:55:05.942836+0000","last_clean":"2023-03-20T18:55:05.942836+0000","last_became_active":"2023-03-20T18:55:01.356978+0000","last_became_peered":"2023-03-20T18:55:01.356978+0000","last_unstale":"2023-03-20T18:55:05.942836+0000","last_undegraded":"2023-03-20T18:55:05.942836+0000","last_fullsized":"2023-03-20T18:55:05.942836+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:12:03.450754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942121+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:55:05.942121+0000","last_peered":"2023-03-20T18:55:05.942121+0000","last_clean":"2023-03-20T18:55:05.942121+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:55:05.942121+0000","last_undegraded":"2023-03-20T18:55:05.942121+0000","last_fullsized":"2023-03-20T18:55:05.942121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:53:17.445676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364026+0000","last_change":"2023-03-20T18:54:59.003766+0000","last_active":"2023-03-20T18:55:03.364026+0000","last_peered":"2023-03-20T18:55:03.364026+0000","last_clean":"2023-03-20T18:55:03.364026+0000","last_became_active":"2023-03-20T18:54:59.002918+0000","last_became_peered":"2023-03-20T18:54:59.002918+0000","last_unstale":"2023-03-20T18:55:03.364026+0000","last_undegraded":"2023-03-20T18:55:03.364026+0000","last_fullsized":"2023-03-20T18:55:03.364026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T01:10:01.900871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364333+0000","last_change":"2023-03-20T18:55:01.356228+0000","last_active":"2023-03-20T18:55:03.364333+0000","last_peered":"2023-03-20T18:55:03.364333+0000","last_clean":"2023-03-20T18:55:03.364333+0000","last_became_active":"2023-03-20T18:55:01.355775+0000","last_became_peered":"2023-03-20T18:55:01.355775+0000","last_unstale":"2023-03-20T18:55:03.364333+0000","last_undegraded":"2023-03-20T18:55:03.364333+0000","last_fullsized":"2023-03-20T18:55:03.364333+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T00:48:50.272984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:05.942222+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:55:05.942222+0000","last_peered":"2023-03-20T18:55:05.942222+0000","last_clean":"2023-03-20T18:55:05.942222+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:55:05.942222+0000","last_undegraded":"2023-03-20T18:55:05.942222+0000","last_fullsized":"2023-03-20T18:55:05.942222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:21:14.194017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-03-20T18:55:03.364561+0000","last_change":"2023-03-20T18:54:58.323785+0000","last_active":"2023-03-20T18:55:03.364561+0000","last_peered":"2023-03-20T18:55:03.364561+0000","last_clean":"2023-03-20T18:55:03.364561+0000","last_became_active":"2023-03-20T18:54:58.323411+0000","last_became_peered":"2023-03-20T18:54:58.323411+0000","last_unstale":"2023-03-20T18:55:03.364561+0000","last_undegraded":"2023-03-20T18:55:03.364561+0000","last_fullsized":"2023-03-20T18:55:03.364561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:32:43.388503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":5,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":4,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":3,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5368,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94366472,"statfs":{"total":96636764160,"available":96631267328,"internally_reserved":0,"allocated":245760,"data_stored":83622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.39800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.46200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.501},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.58899999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607595,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.625,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55000000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 20 18:56:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61599999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.48499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.58699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738412,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.45800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61499999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:56:19.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-20T18:56:21.665 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:56:22.954 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"fsid":"45227b9a-cda4-41c5-b809-1315c0145b2a","created":"2023-03-20T18:52:33.269897+0000","modified":"2023-03-20T18:55:06.378742+0000","last_up_change":"2023-03-20T18:53:00.564459+0000","last_in_change":"2023-03-20T18:52:44.787797+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-20T18:52:59.675326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-20T18:53:11.952974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"slow","create_time":"2023-03-20T18:54:53.677479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":2,"pg_placement_num":2,"pg_placement_num_target":2,"pg_num_target":2,"pg_num_pending":2,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[5,6],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}},{"pool":4,"pool_name":"slow2","create_time":"2023-03-20T18:54:56.637718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":2,"pg_placement_num":2,"pg_placement_num_target":2,"pg_num_target":2,"pg_num_pending":2,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}},{"pool":5,"pool_name":"cache","create_time":"2023-03-20T18:54:59.651373+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":2,"pg_placement_num":2,"pg_placement_num_target":2,"pg_num_target":2,"pg_num_pending":2,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}},{"pool":6,"pool_name":"cache2","create_time":"2023-03-20T18:55:00.776891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":2,"pg_placement_num":2,"pg_placement_num_target":2,"pg_num_target":2,"pg_num_pending":2,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d7dbd9b2-81f9-46ac-bbad-beae282f1a88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6801","nonce":28983}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6802","nonce":28983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6804","nonce":28983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6803","nonce":28983}]},"public_addr":"172.21.15.46:6801/28983","cluster_addr":"172.21.15.46:6802/28983","heartbeat_back_addr":"172.21.15.46:6804/28983","heartbeat_front_addr":"172.21.15.46:6803/28983","state":["exists","up"]},{"osd":1,"uuid":"2b8049e1-db98-4e68-84e1-56c507402b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6805","nonce":29810}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6806","nonce":29810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6808","nonce":29810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6807","nonce":29810}]},"public_addr":"172.21.15.46:6805/29810","cluster_addr":"172.21.15.46:6806/29810","heartbeat_back_addr":"172.21.15.46:6808/29810","heartbeat_front_addr":"172.21.15.46:6807/29810","state":["exists","up"]},{"osd":2,"uuid":"290b5c77-abaa-4e65-ab57-751368bbed68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6809","nonce":29850}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6810","nonce":29850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6812","nonce":29850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.46:6811","nonce":29850}]},"public_addr":"172.21.15.46:6809/29850","cluster_addr":"172.21.15.46:6810/29850","heartbeat_back_addr":"172.21.15.46:6812/29850","heartbeat_front_addr":"172.21.15.46:6811/29850","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:52.453137+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:57.583791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-03-20T18:52:59.409344+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-20T18:56:23.956 INFO:tasks.ceph:Scrubbing osd.0 2023-03-20T18:56:24.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-03-20T18:56:25.113 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-03-20T18:56:26.914 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-03-20T18:56:26.914 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-03-20T18:56:26.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-03-20T18:56:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2023-03-20T18:56:27.735 INFO:tasks.ceph:Scrubbing osd.1 2023-03-20T18:56:27.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-03-20T18:56:27.891 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-03-20T18:56:28.758 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-03-20T18:56:28.758 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-03-20T18:56:28.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-03-20T18:56:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2023-03-20T18:56:29.988 INFO:tasks.ceph:Scrubbing osd.2 2023-03-20T18:56:29.988 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-03-20T18:56:30.138 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-03-20T18:56:30.937 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-03-20T18:56:30.937 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-03-20T18:56:30.938 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-03-20T18:56:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2023-03-20T18:56:32.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:56:32.741 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:56:34.572 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:56:34.572 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2023-03-20T18:56:31.679035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17792,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":283097728,"statfs":{"total":289910292480,"available":289892073472,"internally_reserved":0,"allocated":1679360,"data_stored":1184778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003570"},"pg_stats":[{"pgid":"1.0","version":"14'97","reported_seq":151,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394480+0000","last_change":"2023-03-20T18:53:01.597772+0000","last_active":"2023-03-20T18:55:06.394480+0000","last_peered":"2023-03-20T18:55:06.394480+0000","last_clean":"2023-03-20T18:55:06.394480+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:55:06.394480+0000","last_undegraded":"2023-03-20T18:55:06.394480+0000","last_fullsized":"2023-03-20T18:55:06.394480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:00.564459+0000","last_clean_scrub_stamp":"2023-03-20T18:53:00.564459+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394351+0000","last_change":"2023-03-20T18:53:13.029981+0000","last_active":"2023-03-20T18:55:06.394351+0000","last_peered":"2023-03-20T18:55:06.394351+0000","last_clean":"2023-03-20T18:55:06.394351+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:55:06.394351+0000","last_undegraded":"2023-03-20T18:55:06.394351+0000","last_fullsized":"2023-03-20T18:55:06.394351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.391819+0000","last_change":"2023-03-20T18:53:13.029863+0000","last_active":"2023-03-20T18:55:06.391819+0000","last_peered":"2023-03-20T18:55:06.391819+0000","last_clean":"2023-03-20T18:55:06.391819+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:55:06.391819+0000","last_undegraded":"2023-03-20T18:55:06.391819+0000","last_fullsized":"2023-03-20T18:55:06.391819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:28.539513+0000","last_change":"2023-03-20T18:56:28.539443+0000","last_active":"2023-03-20T18:56:28.539513+0000","last_peered":"2023-03-20T18:56:28.539513+0000","last_clean":"2023-03-20T18:56:28.539513+0000","last_became_active":"2023-03-20T18:54:55.287358+0000","last_became_peered":"2023-03-20T18:54:55.287358+0000","last_unstale":"2023-03-20T18:56:28.539513+0000","last_undegraded":"2023-03-20T18:56:28.539513+0000","last_fullsized":"2023-03-20T18:56:28.539513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:28.539196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:28.539196+0000","last_clean_scrub_stamp":"2023-03-20T18:56:28.539196+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:11:43.794891+0000","scrub_duration":0.0041694619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.392058+0000","last_change":"2023-03-20T18:53:13.030207+0000","last_active":"2023-03-20T18:55:06.392058+0000","last_peered":"2023-03-20T18:55:06.392058+0000","last_clean":"2023-03-20T18:55:06.392058+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:55:06.392058+0000","last_undegraded":"2023-03-20T18:55:06.392058+0000","last_fullsized":"2023-03-20T18:55:06.392058+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394442+0000","last_change":"2023-03-20T18:54:55.286337+0000","last_active":"2023-03-20T18:55:06.394442+0000","last_peered":"2023-03-20T18:55:06.394442+0000","last_clean":"2023-03-20T18:55:06.394442+0000","last_became_active":"2023-03-20T18:54:55.285843+0000","last_became_peered":"2023-03-20T18:54:55.285843+0000","last_unstale":"2023-03-20T18:55:06.394442+0000","last_undegraded":"2023-03-20T18:55:06.394442+0000","last_fullsized":"2023-03-20T18:55:06.394442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:54.262811+0000","last_clean_scrub_stamp":"2023-03-20T18:54:54.262811+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:27.514124+0000","last_change":"2023-03-20T18:56:27.514055+0000","last_active":"2023-03-20T18:56:27.514124+0000","last_peered":"2023-03-20T18:56:27.514124+0000","last_clean":"2023-03-20T18:56:27.514124+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:56:27.514124+0000","last_undegraded":"2023-03-20T18:56:27.514124+0000","last_fullsized":"2023-03-20T18:56:27.514124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:27.513795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:27.513795+0000","last_clean_scrub_stamp":"2023-03-20T18:56:27.513795+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T20:53:22.310069+0000","scrub_duration":0.0037203739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:07.009829+0000","last_change":"2023-03-20T18:55:02.367518+0000","last_active":"2023-03-20T18:55:07.009829+0000","last_peered":"2023-03-20T18:55:07.009829+0000","last_clean":"2023-03-20T18:55:07.009829+0000","last_became_active":"2023-03-20T18:55:02.367219+0000","last_became_peered":"2023-03-20T18:55:02.367219+0000","last_unstale":"2023-03-20T18:55:07.009829+0000","last_undegraded":"2023-03-20T18:55:07.009829+0000","last_fullsized":"2023-03-20T18:55:07.009829+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394415+0000","last_change":"2023-03-20T18:53:13.029389+0000","last_active":"2023-03-20T18:55:06.394415+0000","last_peered":"2023-03-20T18:55:06.394415+0000","last_clean":"2023-03-20T18:55:06.394415+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:55:06.394415+0000","last_undegraded":"2023-03-20T18:55:06.394415+0000","last_fullsized":"2023-03-20T18:55:06.394415+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394311+0000","last_change":"2023-03-20T18:55:02.364036+0000","last_active":"2023-03-20T18:55:06.394311+0000","last_peered":"2023-03-20T18:55:06.394311+0000","last_clean":"2023-03-20T18:55:06.394311+0000","last_became_active":"2023-03-20T18:55:02.363649+0000","last_became_peered":"2023-03-20T18:55:02.363649+0000","last_unstale":"2023-03-20T18:55:06.394311+0000","last_undegraded":"2023-03-20T18:55:06.394311+0000","last_fullsized":"2023-03-20T18:55:06.394311+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:01.337068+0000","last_clean_scrub_stamp":"2023-03-20T18:55:01.337068+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394516+0000","last_change":"2023-03-20T18:53:13.033108+0000","last_active":"2023-03-20T18:55:06.394516+0000","last_peered":"2023-03-20T18:55:06.394516+0000","last_clean":"2023-03-20T18:55:06.394516+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:55:06.394516+0000","last_undegraded":"2023-03-20T18:55:06.394516+0000","last_fullsized":"2023-03-20T18:55:06.394516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394405+0000","last_change":"2023-03-20T18:55:01.357464+0000","last_active":"2023-03-20T18:55:06.394405+0000","last_peered":"2023-03-20T18:55:06.394405+0000","last_clean":"2023-03-20T18:55:06.394405+0000","last_became_active":"2023-03-20T18:55:01.356978+0000","last_became_peered":"2023-03-20T18:55:01.356978+0000","last_unstale":"2023-03-20T18:55:06.394405+0000","last_undegraded":"2023-03-20T18:55:06.394405+0000","last_fullsized":"2023-03-20T18:55:06.394405+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394517+0000","last_change":"2023-03-20T18:53:13.033680+0000","last_active":"2023-03-20T18:55:06.394517+0000","last_peered":"2023-03-20T18:55:06.394517+0000","last_clean":"2023-03-20T18:55:06.394517+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:55:06.394517+0000","last_undegraded":"2023-03-20T18:55:06.394517+0000","last_fullsized":"2023-03-20T18:55:06.394517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:07.009768+0000","last_change":"2023-03-20T18:54:59.003766+0000","last_active":"2023-03-20T18:55:07.009768+0000","last_peered":"2023-03-20T18:55:07.009768+0000","last_clean":"2023-03-20T18:55:07.009768+0000","last_became_active":"2023-03-20T18:54:59.002918+0000","last_became_peered":"2023-03-20T18:54:59.002918+0000","last_unstale":"2023-03-20T18:55:07.009768+0000","last_undegraded":"2023-03-20T18:55:07.009768+0000","last_fullsized":"2023-03-20T18:55:07.009768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.391683+0000","last_change":"2023-03-20T18:55:01.356228+0000","last_active":"2023-03-20T18:55:06.391683+0000","last_peered":"2023-03-20T18:55:06.391683+0000","last_clean":"2023-03-20T18:55:06.391683+0000","last_became_active":"2023-03-20T18:55:01.355775+0000","last_became_peered":"2023-03-20T18:55:01.355775+0000","last_unstale":"2023-03-20T18:55:06.391683+0000","last_undegraded":"2023-03-20T18:55:06.391683+0000","last_fullsized":"2023-03-20T18:55:06.391683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:55:00.327796+0000","last_clean_scrub_stamp":"2023-03-20T18:55:00.327796+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.394599+0000","last_change":"2023-03-20T18:53:13.034520+0000","last_active":"2023-03-20T18:55:06.394599+0000","last_peered":"2023-03-20T18:55:06.394599+0000","last_clean":"2023-03-20T18:55:06.394599+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:55:06.394599+0000","last_undegraded":"2023-03-20T18:55:06.394599+0000","last_fullsized":"2023-03-20T18:55:06.394599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:53:12.006714+0000","last_clean_scrub_stamp":"2023-03-20T18:53:12.006714+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:55:06.391834+0000","last_change":"2023-03-20T18:54:58.323785+0000","last_active":"2023-03-20T18:55:06.391834+0000","last_peered":"2023-03-20T18:55:06.391834+0000","last_clean":"2023-03-20T18:55:06.391834+0000","last_became_active":"2023-03-20T18:54:58.323411+0000","last_became_peered":"2023-03-20T18:54:58.323411+0000","last_unstale":"2023-03-20T18:55:06.391834+0000","last_undegraded":"2023-03-20T18:55:06.391834+0000","last_fullsized":"2023-03-20T18:55:06.391834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:54:57.297037+0000","last_clean_scrub_stamp":"2023-03-20T18:54:57.297037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":5,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":4,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":3,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5368,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94366472,"statfs":{"total":96636764160,"available":96631267328,"internally_reserved":0,"allocated":245760,"data_stored":83622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.39800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.46200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42199999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.625,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.626},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55000000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.5}]},{"osd":2,"last update":"Mon Mar 20 18:56:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61599999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.48499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.60899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.45800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61499999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:56:34.944 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-03-20T18:53:00.564459+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=0, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:36.548 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:36.548 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:37.742 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:39.064 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-03-20T18:54:54.262811+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=54, tm_sec=54, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:39.064 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2023-03-20T18:55:01.337068+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=55, tm_sec=1, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:39.064 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:39.064 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2023-03-20T18:55:01.337068+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=55, tm_sec=1, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:39.064 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.330 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2023-03-20T18:55:00.327796+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=55, tm_sec=0, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.330 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.331 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-03-20T18:54:57.297037+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=54, tm_sec=57, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.331 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2023-03-20T18:55:00.327796+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=55, tm_sec=0, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.331 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-03-20T18:53:12.006714+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=53, tm_sec=12, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.331 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-03-20T18:54:57.297037+0000 time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=54, tm_sec=57, tm_wday=0, tm_yday=79, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=3, tm_mday=20, tm_hour=18, tm_min=56, tm_sec=22, tm_wday=0, tm_yday=79, tm_isdst=0) 2023-03-20T18:56:40.331 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-03-20T18:57:00.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-20T18:57:04.131 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-03-20T18:57:06.057 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-03-20T18:57:08.341 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2023-03-20T18:57:03.688634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17792,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":283097728,"statfs":{"total":289910292480,"available":289892073472,"internally_reserved":0,"allocated":1679360,"data_stored":1184778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"1.0","version":"29'99","reported_seq":163,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:30.944479+0000","last_change":"2023-03-20T18:56:30.944425+0000","last_active":"2023-03-20T18:56:30.944479+0000","last_peered":"2023-03-20T18:56:30.944479+0000","last_clean":"2023-03-20T18:56:30.944479+0000","last_became_active":"2023-03-20T18:53:01.595861+0000","last_became_peered":"2023-03-20T18:53:01.595861+0000","last_unstale":"2023-03-20T18:56:30.944479+0000","last_undegraded":"2023-03-20T18:56:30.944479+0000","last_fullsized":"2023-03-20T18:56:30.944479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'99","last_scrub_stamp":"2023-03-20T18:56:30.944215+0000","last_deep_scrub":"29'99","last_deep_scrub_stamp":"2023-03-20T18:56:30.944215+0000","last_clean_scrub_stamp":"2023-03-20T18:56:30.944215+0000","objects_scrubbed":2,"log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T22:57:09.494171+0000","scrub_duration":0.011007543999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:42.968634+0000","last_change":"2023-03-20T18:56:42.968561+0000","last_active":"2023-03-20T18:56:42.968634+0000","last_peered":"2023-03-20T18:56:42.968634+0000","last_clean":"2023-03-20T18:56:42.968634+0000","last_became_active":"2023-03-20T18:53:13.029150+0000","last_became_peered":"2023-03-20T18:53:13.029150+0000","last_unstale":"2023-03-20T18:56:42.968634+0000","last_undegraded":"2023-03-20T18:56:42.968634+0000","last_fullsized":"2023-03-20T18:56:42.968634+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:42.968284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:42.968284+0000","last_clean_scrub_stamp":"2023-03-20T18:56:42.968284+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T05:15:34.214878+0000","scrub_duration":0.0046899869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:37.611006+0000","last_change":"2023-03-20T18:56:37.610954+0000","last_active":"2023-03-20T18:56:37.611006+0000","last_peered":"2023-03-20T18:56:37.611006+0000","last_clean":"2023-03-20T18:56:37.611006+0000","last_became_active":"2023-03-20T18:53:13.029378+0000","last_became_peered":"2023-03-20T18:53:13.029378+0000","last_unstale":"2023-03-20T18:56:37.611006+0000","last_undegraded":"2023-03-20T18:56:37.611006+0000","last_fullsized":"2023-03-20T18:56:37.611006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:37.610689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:37.610689+0000","last_clean_scrub_stamp":"2023-03-20T18:56:37.610689+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T02:18:59.502766+0000","scrub_duration":0.0048188079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:28.539513+0000","last_change":"2023-03-20T18:56:28.539443+0000","last_active":"2023-03-20T18:56:28.539513+0000","last_peered":"2023-03-20T18:56:28.539513+0000","last_clean":"2023-03-20T18:56:28.539513+0000","last_became_active":"2023-03-20T18:54:55.287358+0000","last_became_peered":"2023-03-20T18:54:55.287358+0000","last_unstale":"2023-03-20T18:56:28.539513+0000","last_undegraded":"2023-03-20T18:56:28.539513+0000","last_fullsized":"2023-03-20T18:56:28.539513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:28.539196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:28.539196+0000","last_clean_scrub_stamp":"2023-03-20T18:56:28.539196+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:11:43.794891+0000","scrub_duration":0.0041694619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:31.738231+0000","last_change":"2023-03-20T18:56:31.738167+0000","last_active":"2023-03-20T18:56:31.738231+0000","last_peered":"2023-03-20T18:56:31.738231+0000","last_clean":"2023-03-20T18:56:31.738231+0000","last_became_active":"2023-03-20T18:53:13.029454+0000","last_became_peered":"2023-03-20T18:53:13.029454+0000","last_unstale":"2023-03-20T18:56:31.738231+0000","last_undegraded":"2023-03-20T18:56:31.738231+0000","last_fullsized":"2023-03-20T18:56:31.738231+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:31.737933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:31.737933+0000","last_clean_scrub_stamp":"2023-03-20T18:56:31.737933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T01:39:29.702939+0000","scrub_duration":0.0048215949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:53.055847+0000","last_change":"2023-03-20T18:56:53.055796+0000","last_active":"2023-03-20T18:56:53.055847+0000","last_peered":"2023-03-20T18:56:53.055847+0000","last_clean":"2023-03-20T18:56:53.055847+0000","last_became_active":"2023-03-20T18:54:55.285843+0000","last_became_peered":"2023-03-20T18:54:55.285843+0000","last_unstale":"2023-03-20T18:56:53.055847+0000","last_undegraded":"2023-03-20T18:56:53.055847+0000","last_fullsized":"2023-03-20T18:56:53.055847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:53.055541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:53.055541+0000","last_clean_scrub_stamp":"2023-03-20T18:56:53.055541+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T23:23:53.321278+0000","scrub_duration":0.00489005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:27.514124+0000","last_change":"2023-03-20T18:56:27.514055+0000","last_active":"2023-03-20T18:56:27.514124+0000","last_peered":"2023-03-20T18:56:27.514124+0000","last_clean":"2023-03-20T18:56:27.514124+0000","last_became_active":"2023-03-20T18:53:13.030744+0000","last_became_peered":"2023-03-20T18:53:13.030744+0000","last_unstale":"2023-03-20T18:56:27.514124+0000","last_undegraded":"2023-03-20T18:56:27.514124+0000","last_fullsized":"2023-03-20T18:56:27.514124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:27.513795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:27.513795+0000","last_clean_scrub_stamp":"2023-03-20T18:56:27.513795+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T20:53:22.310069+0000","scrub_duration":0.0037203739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:32.546427+0000","last_change":"2023-03-20T18:56:32.546388+0000","last_active":"2023-03-20T18:56:32.546427+0000","last_peered":"2023-03-20T18:56:32.546427+0000","last_clean":"2023-03-20T18:56:32.546427+0000","last_became_active":"2023-03-20T18:55:02.367219+0000","last_became_peered":"2023-03-20T18:55:02.367219+0000","last_unstale":"2023-03-20T18:56:32.546427+0000","last_undegraded":"2023-03-20T18:56:32.546427+0000","last_fullsized":"2023-03-20T18:56:32.546427+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:32.546279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:32.546279+0000","last_clean_scrub_stamp":"2023-03-20T18:56:32.546279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T18:57:04.955598+0000","scrub_duration":0.0031975049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:52.103540+0000","last_change":"2023-03-20T18:56:52.103450+0000","last_active":"2023-03-20T18:56:52.103540+0000","last_peered":"2023-03-20T18:56:52.103540+0000","last_clean":"2023-03-20T18:56:52.103540+0000","last_became_active":"2023-03-20T18:53:13.029069+0000","last_became_peered":"2023-03-20T18:53:13.029069+0000","last_unstale":"2023-03-20T18:56:52.103540+0000","last_undegraded":"2023-03-20T18:56:52.103540+0000","last_fullsized":"2023-03-20T18:56:52.103540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:52.103079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:52.103079+0000","last_clean_scrub_stamp":"2023-03-20T18:56:52.103079+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T01:19:53.722210+0000","scrub_duration":0.0047584690000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:57:00.046214+0000","last_change":"2023-03-20T18:57:00.046134+0000","last_active":"2023-03-20T18:57:00.046214+0000","last_peered":"2023-03-20T18:57:00.046214+0000","last_clean":"2023-03-20T18:57:00.046214+0000","last_became_active":"2023-03-20T18:55:02.363649+0000","last_became_peered":"2023-03-20T18:55:02.363649+0000","last_unstale":"2023-03-20T18:57:00.046214+0000","last_undegraded":"2023-03-20T18:57:00.046214+0000","last_fullsized":"2023-03-20T18:57:00.046214+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:57:00.045764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:57:00.045764+0000","last_clean_scrub_stamp":"2023-03-20T18:57:00.045764+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T02:58:37.120460+0000","scrub_duration":0.0045264770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:37.951721+0000","last_change":"2023-03-20T18:56:37.951664+0000","last_active":"2023-03-20T18:56:37.951721+0000","last_peered":"2023-03-20T18:56:37.951721+0000","last_clean":"2023-03-20T18:56:37.951721+0000","last_became_active":"2023-03-20T18:53:13.032748+0000","last_became_peered":"2023-03-20T18:53:13.032748+0000","last_unstale":"2023-03-20T18:56:37.951721+0000","last_undegraded":"2023-03-20T18:56:37.951721+0000","last_fullsized":"2023-03-20T18:56:37.951721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:37.951402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:37.951402+0000","last_clean_scrub_stamp":"2023-03-20T18:56:37.951402+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T02:59:22.984772+0000","scrub_duration":0.0050664539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:59.080307+0000","last_change":"2023-03-20T18:56:59.080257+0000","last_active":"2023-03-20T18:56:59.080307+0000","last_peered":"2023-03-20T18:56:59.080307+0000","last_clean":"2023-03-20T18:56:59.080307+0000","last_became_active":"2023-03-20T18:55:01.356978+0000","last_became_peered":"2023-03-20T18:55:01.356978+0000","last_unstale":"2023-03-20T18:56:59.080307+0000","last_undegraded":"2023-03-20T18:56:59.080307+0000","last_fullsized":"2023-03-20T18:56:59.080307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:59.080095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:59.080095+0000","last_clean_scrub_stamp":"2023-03-20T18:56:59.080095+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T21:29:04.330181+0000","scrub_duration":0.004001999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:49.042708+0000","last_change":"2023-03-20T18:56:49.042608+0000","last_active":"2023-03-20T18:56:49.042708+0000","last_peered":"2023-03-20T18:56:49.042708+0000","last_clean":"2023-03-20T18:56:49.042708+0000","last_became_active":"2023-03-20T18:53:13.033042+0000","last_became_peered":"2023-03-20T18:53:13.033042+0000","last_unstale":"2023-03-20T18:56:49.042708+0000","last_undegraded":"2023-03-20T18:56:49.042708+0000","last_fullsized":"2023-03-20T18:56:49.042708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:49.042217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:49.042217+0000","last_clean_scrub_stamp":"2023-03-20T18:56:49.042217+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T04:20:38.634720+0000","scrub_duration":0.0057208270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:29.539785+0000","last_change":"2023-03-20T18:56:29.539686+0000","last_active":"2023-03-20T18:56:29.539785+0000","last_peered":"2023-03-20T18:56:29.539785+0000","last_clean":"2023-03-20T18:56:29.539785+0000","last_became_active":"2023-03-20T18:54:59.002918+0000","last_became_peered":"2023-03-20T18:54:59.002918+0000","last_unstale":"2023-03-20T18:56:29.539785+0000","last_undegraded":"2023-03-20T18:56:29.539785+0000","last_fullsized":"2023-03-20T18:56:29.539785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:29.539433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:29.539433+0000","last_clean_scrub_stamp":"2023-03-20T18:56:29.539433+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T19:57:18.334590+0000","scrub_duration":0.0037644330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:42.644747+0000","last_change":"2023-03-20T18:56:42.644656+0000","last_active":"2023-03-20T18:56:42.644747+0000","last_peered":"2023-03-20T18:56:42.644747+0000","last_clean":"2023-03-20T18:56:42.644747+0000","last_became_active":"2023-03-20T18:55:01.355775+0000","last_became_peered":"2023-03-20T18:55:01.355775+0000","last_unstale":"2023-03-20T18:56:42.644747+0000","last_undegraded":"2023-03-20T18:56:42.644747+0000","last_fullsized":"2023-03-20T18:56:42.644747+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:42.644419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:42.644419+0000","last_clean_scrub_stamp":"2023-03-20T18:56:42.644419+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T22:41:44.693619+0000","scrub_duration":0.004736227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:41.988820+0000","last_change":"2023-03-20T18:56:41.988760+0000","last_active":"2023-03-20T18:56:41.988820+0000","last_peered":"2023-03-20T18:56:41.988820+0000","last_clean":"2023-03-20T18:56:41.988820+0000","last_became_active":"2023-03-20T18:53:13.034220+0000","last_became_peered":"2023-03-20T18:53:13.034220+0000","last_unstale":"2023-03-20T18:56:41.988820+0000","last_undegraded":"2023-03-20T18:56:41.988820+0000","last_fullsized":"2023-03-20T18:56:41.988820+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:41.988518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:41.988518+0000","last_clean_scrub_stamp":"2023-03-20T18:56:41.988518+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-21T22:00:31.691925+0000","scrub_duration":0.0046350230000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-03-20T18:56:41.689267+0000","last_change":"2023-03-20T18:56:41.689174+0000","last_active":"2023-03-20T18:56:41.689267+0000","last_peered":"2023-03-20T18:56:41.689267+0000","last_clean":"2023-03-20T18:56:41.689267+0000","last_became_active":"2023-03-20T18:54:58.323411+0000","last_became_peered":"2023-03-20T18:54:58.323411+0000","last_unstale":"2023-03-20T18:56:41.689267+0000","last_undegraded":"2023-03-20T18:56:41.689267+0000","last_fullsized":"2023-03-20T18:56:41.689267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-20T18:56:41.688919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-20T18:56:41.688919+0000","last_clean_scrub_stamp":"2023-03-20T18:56:41.688919+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-03-22T03:57:13.488600+0000","scrub_duration":0.0046313889999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":5,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":4,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":3,"num_pg":2,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5368,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94366472,"statfs":{"total":96636764160,"available":96631267328,"internally_reserved":0,"allocated":245760,"data_stored":83622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.39800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.55400000000000005},"min":{"1min":0.46200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 20 18:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65700000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.625}]}]},{"osd":1,"up_from":12,"seq":51539607604,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 20 18:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.625,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55000000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Mar 20 18:56:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61599999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.48499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.63600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6212,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94365628,"statfs":{"total":96636764160,"available":96630403072,"internally_reserved":0,"allocated":716800,"data_stored":550578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.435,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.45800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Mar 20 18:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61499999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.48699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67500000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-20T18:57:10.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-03-20T18:57:10.863 INFO:teuthology.misc:Shutting down mds daemons... 2023-03-20T18:57:12.544 INFO:teuthology.misc:Shutting down osd daemons... 2023-03-20T18:57:12.546 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-03-20T18:57:12.546 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:12.598 INFO:tasks.ceph.osd.0:Stopped 2023-03-20T18:57:14.053 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-03-20T18:57:14.053 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:14.136 INFO:tasks.ceph.osd.1:Stopped 2023-03-20T18:57:15.127 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-03-20T18:57:16.633 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:16.634 INFO:tasks.ceph.osd.2:Stopped 2023-03-20T18:57:16.634 INFO:teuthology.misc:Shutting down mgr daemons... 2023-03-20T18:57:16.634 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-03-20T18:57:16.634 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:16.660 INFO:tasks.ceph.mgr.x:Stopped 2023-03-20T18:57:19.780 INFO:teuthology.misc:Shutting down mon daemons... 2023-03-20T18:57:19.781 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-03-20T18:57:19.781 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:19.799 INFO:tasks.ceph.mon.a:Stopped 2023-03-20T18:57:21.616 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-03-20T18:57:21.617 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:21.635 INFO:tasks.ceph.mon.b:Stopped 2023-03-20T18:57:23.258 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-03-20T18:57:23.258 INFO:teuthology.orchestra.run:waiting for 300 2023-03-20T18:57:23.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-03-20T18:57:23.276 INFO:tasks.ceph.mon.c:Stopped 2023-03-20T18:57:24.799 INFO:tasks.ceph:Checking cluster log for badness... 2023-03-20T18:57:24.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-03-20T18:57:24.824 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:57:25.621 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-03-20T18:57:25.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:57:27.087 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-03-20T18:57:27.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:57:29.380 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-03-20T18:57:29.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-03-20T18:57:29.558 INFO:tasks.ceph:Archiving mon data... 2023-03-20T18:57:30.982 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/data/mon.a.tgz 2023-03-20T18:57:30.983 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-03-20T18:57:30.990 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.tZbBV0Zj1A 2023-03-20T18:57:30.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.tZbBV0Zj1A 2023-03-20T18:57:33.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.tZbBV0Zj1A 2023-03-20T18:57:36.001 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.tZbBV0Zj1A is 198KB 2023-03-20T18:57:39.398 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.tZbBV0Zj1A 2023-03-20T18:57:41.141 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/data/mon.b.tgz 2023-03-20T18:57:44.808 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-03-20T18:57:44.815 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.vwbosaLovD 2023-03-20T18:57:46.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vwbosaLovD 2023-03-20T18:57:46.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.vwbosaLovD 2023-03-20T18:57:48.137 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.vwbosaLovD is 262KB 2023-03-20T18:57:49.748 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.vwbosaLovD 2023-03-20T18:57:52.094 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/data/mon.c.tgz 2023-03-20T18:57:53.445 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-03-20T18:57:53.451 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.D33XmXPsZh 2023-03-20T18:57:54.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.D33XmXPsZh 2023-03-20T18:57:54.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.D33XmXPsZh 2023-03-20T18:57:56.737 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.D33XmXPsZh is 262KB 2023-03-20T18:57:58.520 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.D33XmXPsZh 2023-03-20T18:58:00.987 INFO:tasks.ceph:Cleaning ceph cluster... 2023-03-20T18:58:01.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-03-20T18:58:01.651 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-03-20T18:58:03.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-03-20T18:58:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-03-20T18:58:05.988 INFO:tasks.ceph:Archiving crash dumps... 2023-03-20T18:58:07.476 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/remote/smithi046/crash 2023-03-20T18:58:08.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2023-03-20T18:58:08.931 INFO:tasks.ceph:Compressing logs... 2023-03-20T18:58:10.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-03-20T18:58:11.108 INFO:tasks.ceph:Archiving logs... 2023-03-20T18:58:12.504 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/remote/smithi046/log 2023-03-20T18:58:14.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/log/ceph -- . 2023-03-20T18:58:14.959 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-03-20T18:58:17.727 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-03-20T18:58:19.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-03-20T18:58:20.009 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-20T18:58:20.823 INFO:teuthology.task.clock:Checking final clock skew... 2023-03-20T18:58:21.309 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-20T18:58:21.330 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2023-03-20T18:58:22.235 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2023-03-20T18:58:24.022 INFO:teuthology.orchestra.run.smithi046.stdout:*hv01.front.sepi 44.4.53.2 2 u 42 64 377 0.082 8.327 2.667 2023-03-20T18:58:24.022 INFO:teuthology.orchestra.run.smithi046.stdout:+hv02.front.sepi 63.231.80.2 3 u 41 64 377 0.073 5.201 1.762 2023-03-20T18:58:24.022 INFO:teuthology.orchestra.run.smithi046.stdout:+hv03.front.sepi 152.44.36.235 3 u 43 64 377 0.103 0.651 2.578 2023-03-20T18:58:24.022 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 0.000 0.000 2023-03-20T18:58:24.024 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-03-20T18:58:24.040 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-03-20T18:58:26.706 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-03-20T18:58:26.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-03-20T18:58:27.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-03-20T18:58:29.505 INFO:teuthology.task.internal:Duration was 1065.113405 seconds 2023-03-20T18:58:30.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-03-20T18:58:30.828 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-03-20T18:58:31.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-03-20T18:58:31.492 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-03-20T18:58:33.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2023-03-20T18:58:33.193 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-03-20T18:58:33.208 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-03-20T18:58:34.711 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-03-20T18:58:34.748 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-03-20T18:58:35.854 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-03-20T18:58:35.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-03-20T18:58:37.454 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-03-20T18:58:38.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-03-20T18:58:38.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-03-20T18:58:39.180 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-03-20T18:58:40.671 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2023-03-20T18:58:41.903 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2023-03-20T18:58:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:58:43.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-03-20T18:58:43.144 INFO:teuthology.task.internal:Transferring archived files... 2023-03-20T18:58:44.253 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146/remote/smithi046 2023-03-20T18:58:44.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-03-20T18:58:44.292 INFO:teuthology.task.internal:Removing archive directory... 2023-03-20T18:58:45.363 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2023-03-20T18:58:45.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-03-20T18:58:46.151 INFO:teuthology.task.internal:Not uploading archives. 2023-03-20T18:58:47.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-03-20T18:58:47.621 INFO:teuthology.task.internal:Tidying up after the test... 2023-03-20T18:58:48.481 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-03-20T18:58:48.490 INFO:teuthology.orchestra.run.smithi046.stdout: 924115 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 20 18:58 /home/ubuntu/cephtest 2023-03-20T18:58:49.173 INFO:teuthology.orchestra.run.smithi046.stdout: 1068115 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 20 18:53 /home/ubuntu/cephtest/mnt.0 2023-03-20T18:58:49.174 INFO:teuthology.orchestra.run.smithi046.stdout: 1068116 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 20 18:54 /home/ubuntu/cephtest/mnt.0/client.0 2023-03-20T18:58:49.174 INFO:teuthology.orchestra.run.smithi046.stdout: 1081524 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 20 18:54 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-03-20T18:58:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-20T18:58:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-03-20T18:58:49.175 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/task/internal/__init__.py", line 54, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-03-20T18:58:49.181 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-03-20T18:58:49.995 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi046 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-03-20T18:58:50.632 INFO:teuthology.nuke:Checking targets against current locks 2023-03-20T18:58:50.664 DEBUG:teuthology.nuke:shortname: smithi046 2023-03-20T18:58:54.987 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-20T18:58:55.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213146', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-20 18:28:40.410570', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-03-20T18:58:57.918 INFO:teuthology.orchestra.console:Power off smithi046 2023-03-20T18:58:59.614 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-03-20T18:58:59.637 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-03-20T18:59:05.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-03-20T18:59:10.722 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-03-20T18:59:11.616 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2023-03-20T18:59:12.535 INFO:teuthology.run:Summary data: description: rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} duration: 1065.113404750824 failure_reason: 'Command failed (workunit test cephtool/test.sh) on smithi046 with status 22: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=633652ad6fdb406c890cd795dd1937ca status: fail success: false 2023-03-20T18:59:15.059 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-20T18:59:15.159 INFO:teuthology.run:FAIL