2023-09-29T22:49:06.950 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T22:49:06.950 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --description rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505 -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/orig.config.yaml 2023-09-29T22:49:06.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T22:49:07.059 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false job_id: '7406505' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T22:49:07.059 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-29T22:49:07.059 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-29T22:49:07.060 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T22:49:07.061 INFO:teuthology.task.internal:Checking packages... 2023-09-29T22:49:07.080 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-29T22:49:07.080 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T22:49:07.080 INFO:teuthology.packaging:ref: None 2023-09-29T22:49:07.080 INFO:teuthology.packaging:tag: None 2023-09-29T22:49:07.080 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:49:07.080 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T22:49:07.081 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:49:07.301 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165-g93dce5e1-1focal 2023-09-29T22:49:07.301 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T22:49:07.310 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T22:49:07.310 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T22:49:07.336 INFO:teuthology.task.internal:Saving configuration 2023-09-29T22:49:07.346 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T22:49:07.358 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T22:49:07.382 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:43:15.207984', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-29T22:49:07.382 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T22:49:07.390 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-29T22:49:07.390 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T22:49:07.437 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1919d2aee0>, signals=[15]) 2023-09-29T22:49:07.438 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T22:49:07.446 INFO:teuthology.task.internal:Opening connections... 2023-09-29T22:49:07.446 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T22:49:07.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:49:07.498 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T22:49:07.506 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2023-09-29T22:49:07.617 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2023-09-29T22:49:07.617 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2023-09-29T22:49:07.623 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2023-09-29T22:49:07.623 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-29T22:49:07.623 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2023-09-29T22:49:07.623 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="20.04" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=focal 2023-09-29T22:49:07.624 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=focal 2023-09-29T22:49:07.625 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2023-09-29T22:49:07.648 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T22:49:07.656 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T22:49:07.674 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T22:49:07.674 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2023-09-29T22:49:07.680 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T22:49:07.700 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T22:49:07.700 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2023-09-29T22:49:07.833 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T22:49:07.865 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T22:49:07.878 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T22:49:07.878 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T22:49:07.879 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T22:49:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:49:07.884 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-09-29T22:49:07.890 INFO:teuthology.orchestra.run.smithi064.stdout:5.4.0-124-generic 2023-09-29T22:49:07.891 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.4.0-124-generic 2023-09-29T22:49:07.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-09-29T22:49:08.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-09-29T22:49:08.152 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T22:49:08.152 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-29T22:49:08.152 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-29T22:49:08.235 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-29T22:49:08.532 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-29T22:49:08.617 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-29T22:49:08.657 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-29T22:49:08.660 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-29T22:49:08.661 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-29T22:49:08.676 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-29T22:49:08.676 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-29T22:49:08.677 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-29T22:49:08.681 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T22:49:08.681 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-29T22:49:08.687 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-29T22:49:08.692 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-29T22:49:08.694 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-29T22:49:08.694 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-29T22:49:08.694 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-29T22:49:08.709 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-29T22:49:08.709 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-29T22:49:08.791 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-29T22:49:08.801 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-29T22:49:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-29T22:49:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-29T22:49:08.836 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-29T22:49:08.837 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-29T22:49:08.839 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T22:49:08.839 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-29T22:49:08.847 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-29T22:49:08.852 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-29T22:49:08.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-29T22:49:08.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-29T22:49:08.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-29T22:49:08.872 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-29T22:49:08.872 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-29T22:49:08.872 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-29T22:49:08.873 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-29T22:49:08.873 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-29T22:49:08.891 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-29T22:49:08.892 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-29T22:49:08.892 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-29T22:49:12.710 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 17.4 MB in 3s (6,688 kB/s) 2023-09-29T22:49:14.153 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:49:14.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T22:49:14.311 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:49:14.520 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-09-29T22:49:14.520 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-09-29T22:49:14.715 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-09-29T22:49:14.715 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T22:49:14.715 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-29T22:49:14.715 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-29T22:49:14.716 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-09-29T22:49:14.716 INFO:teuthology.orchestra.run.smithi064.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-29T22:49:14.716 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-09-29T22:49:14.716 INFO:teuthology.orchestra.run.smithi064.stdout: thermald 2023-09-29T22:49:14.785 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-09-29T22:49:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T22:49:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-29T22:49:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-29T22:49:14.786 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-09-29T22:49:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-29T22:49:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T22:49:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 77.1 MB of archives. 2023-09-29T22:49:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-29T22:49:14.845 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-29T22:49:15.249 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-29T22:49:15.352 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-29T22:49:15.925 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-29T22:49:15.926 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-29T22:49:15.926 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-29T22:49:16.084 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-29T22:49:16.100 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-29T22:49:16.798 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 77.1 MB in 1s (58.9 MB/s) 2023-09-29T22:49:16.990 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-29T22:49:18.683 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-29T22:49:18.688 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T22:49:18.771 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:20.737 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-29T22:49:20.752 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T22:49:20.853 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:22.090 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-29T22:49:22.105 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T22:49:22.148 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:26.861 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T22:49:26.986 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T22:49:27.353 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T22:49:27.481 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T22:49:27.809 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-29T22:49:27.826 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-29T22:49:27.867 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T22:49:33.817 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-29T22:49:33.841 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T22:49:33.883 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T22:49:36.926 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T22:49:37.262 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:38.489 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T22:49:38.615 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:39.805 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-29T22:49:39.805 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-29T22:49:39.950 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:40.093 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:41.319 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-29T22:49:41.770 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-29T22:49:42.622 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-29T22:49:42.748 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T22:49:42.819 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-29T22:49:42.820 INFO:teuthology.orchestra.run.smithi064.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-29T22:50:05.852 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-29T22:50:05.853 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub' 2023-09-29T22:50:05.858 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T22:50:05.879 INFO:teuthology.orchestra.run.smithi064.stdout:Generating grub configuration file ... 2023-09-29T22:50:06.168 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T22:50:06.188 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T22:50:06.549 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T22:50:06.560 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T22:50:08.607 INFO:teuthology.orchestra.run.smithi064.stdout:done 2023-09-29T22:50:09.039 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-09-29T22:50:09.089 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.4.0.163.160 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-09-29T22:50:09.090 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-09-29T22:50:09.090 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-29T22:50:09.090 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T22:50:09.090 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T22:50:09.091 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-29T22:50:09.091 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T22:50:09.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-09-29T22:50:09.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-09-29T22:50:09.295 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T22:50:09.300 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T22:50:09.305 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T22:50:09.317 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T22:50:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:50:11.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T22:50:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:50:11.389 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-09-29T22:50:11.390 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-09-29T22:50:11.606 INFO:teuthology.orchestra.run.smithi064.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-29T22:50:11.606 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T22:50:11.608 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-09-29T22:50:11.629 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-09-29T22:50:11.629 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-09-29T22:50:11.629 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-09-29T22:50:11.629 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.4.0.163.160 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-09-29T22:50:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-09-29T22:50:11.631 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-09-29T22:50:11.677 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.ztMmst32Qj 2023-09-29T22:50:11.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ztMmst32Qj 2023-09-29T22:50:11.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.ztMmst32Qj 2023-09-29T22:50:11.752 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.ztMmst32Qj is 9KB 2023-09-29T22:50:11.766 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.ztMmst32Qj 2023-09-29T22:50:11.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-29T22:50:11.827 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:50:11.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-29T22:50:11.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-29T22:50:11.860 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-29T22:50:11.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo update-grub 2023-09-29T22:50:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub' 2023-09-29T22:50:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T22:50:13.161 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2023-09-29T22:50:13.407 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T22:50:13.419 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T22:50:13.759 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T22:50:13.772 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T22:50:14.706 INFO:teuthology.orchestra.run.smithi064.stderr:done 2023-09-29T22:50:14.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2023-09-29T22:50:44.741 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-29T22:50:44.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:50:44.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:51:03.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-09-29T22:51:12.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:51:12.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:51:15.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-09-29T22:51:27.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:51:27.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:51:45.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-09-29T22:52:00.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:52:00.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:52:00.884 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-09-29T22:52:01.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:52:01.447 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-163-generic"... 2023-09-29T22:52:01.447 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-09-29T22:52:01.453 INFO:teuthology.orchestra.run.smithi064.stdout:5.4.0-163-generic 2023-09-29T22:52:01.453 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-29T22:52:01.454 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T22:52:01.454 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-29T22:52:02.455 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-29T22:52:02.455 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T22:52:02.546 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2023-09-29T22:52:02.568 DEBUG:teuthology.parallel:result is None 2023-09-29T22:52:02.569 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T22:52:02.575 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T22:52:02.576 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T22:52:02.583 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T22:52:02.609 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T22:52:02.645 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T22:52:02.645 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T22:52:02.665 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T22:52:02.678 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T22:52:02.679 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T22:52:02.696 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:52:02.707 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:52:02.709 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T22:52:02.720 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T22:52:02.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T22:52:02.735 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T22:52:02.748 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T22:52:02.748 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T22:52:02.784 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T22:52:02.791 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T22:52:02.798 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:52:02.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T22:52:02.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2023-09-29T22:52:02.924 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T22:52:02.931 INFO:teuthology.task.internal:Starting timer... 2023-09-29T22:52:02.932 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T22:52:02.953 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T22:52:02.986 DEBUG:teuthology.task.selinux:Excluding smithi064: OS 'ubuntu' does not support SELinux 2023-09-29T22:52:02.986 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T22:52:02.986 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-29T22:52:02.986 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T22:52:02.987 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T22:52:03.020 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-29T22:52:03.020 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T22:52:03.036 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T22:52:03.037 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T22:57:42.904 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com')] 2023-09-29T22:57:42.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:57:42.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:57:42.955 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-09-29T22:57:43.080 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-09-29T22:57:43.080 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T22:57:43.092 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T22:57:43.092 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T22:57:43.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T22:57:43.130 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-29T22:57:43.131 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Command line: ntpd -gq 2023-09-29T22:57:43.132 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: proto: precision = 0.076 usec (-24) 2023-09-29T22:57:43.132 INFO:teuthology.orchestra.run.smithi064.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T22:57:43.132 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T22:57:43.133 INFO:teuthology.orchestra.run.smithi064.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: restrict ::: KOD does nothing without LIMITED. 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen and drop on 0 v6wildcard [::]:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen normally on 2 lo 127.0.0.1:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen normally on 3 enp3s0f1 172.21.15.64:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen normally on 4 lo [::1]:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8df%5]:123 2023-09-29T22:57:43.134 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:43 ntpd[22700]: Listening on routing socket on fd #22 for interface updates 2023-09-29T22:57:45.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:45.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:45.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:45.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:45.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:45.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:45 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:46.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:46.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:46.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:46.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:46.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:46.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:46 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:47.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:47.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:47 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:48 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:49.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:49.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:49.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:49.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:49.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:49.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:49 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:50.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:50.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:50.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:50.134 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:50.134 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:50 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:51.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:51.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:51.131 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T22:57:51.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T22:57:51.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T22:57:51.132 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T22:57:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:29 Sep 22:57:51 ntpd[22700]: ntpd: time slew +0.001525 s 2023-09-29T22:57:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:ntpd: time slew +0.001525s 2023-09-29T22:57:51.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-29T22:57:51.133 INFO:teuthology.orchestra.run.smithi064.stderr:29 Sep 22:57:51 ntpd[22700]: can't open /var/log/ntpstats/loopstats.20230929: Permission denied 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T22:57:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T22:57:51.197 INFO:teuthology.run_tasks:Running task install... 2023-09-29T22:57:51.205 DEBUG:teuthology.task.install:project ceph 2023-09-29T22:57:51.206 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-29T22:57:51.206 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-29T22:57:51.206 INFO:teuthology.task.install:Using flavor: default 2023-09-29T22:57:51.212 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-29T22:57:51.212 INFO:teuthology.task.install:extra packages: [] 2023-09-29T22:57:51.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-key list | grep Ceph 2023-09-29T22:57:51.340 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-29T22:57:51.403 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-29T22:57:51.403 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph.com (release key) 2023-09-29T22:57:51.403 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-29T22:57:51.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T22:57:51.547 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-29T22:57:51.547 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-29T22:57:51.720 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:57:51.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-29T22:57:51.737 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-09-29T22:57:51.922 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T22:57:51.923 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T22:57:51.935 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T22:57:51.954 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T22:57:51.978 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-29T22:57:51.999 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-29T22:57:52.022 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-29T22:57:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-29T22:57:52.092 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-29T22:57:52.593 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 28.4 kB in 1s (43.9 kB/s) 2023-09-29T22:57:53.652 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:57:53.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-29T22:57:53.748 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-09-29T22:57:53.956 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-09-29T22:57:53.956 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-09-29T22:57:54.138 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-09-29T22:57:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-29T22:57:54.139 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-29T22:57:54.139 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-09-29T22:57:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-29T22:57:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-29T22:57:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-29T22:57:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-29T22:57:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-29T22:57:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-29T22:57:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: socat xmlstarlet 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-29T22:57:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-09-29T22:57:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: btrfs-tools 2023-09-29T22:57:54.263 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-09-29T22:57:54.263 INFO:teuthology.orchestra.run.smithi064.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-29T22:57:54.264 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-29T22:57:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-29T22:57:54.266 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-09-29T22:57:54.267 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 librbd1 2023-09-29T22:57:54.329 INFO:teuthology.orchestra.run.smithi064.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-29T22:57:54.329 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 209 MB of archives. 2023-09-29T22:57:54.329 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-29T22:57:54.330 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-29T22:57:54.412 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-29T22:57:54.413 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-29T22:57:54.416 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-29T22:57:54.422 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-29T22:57:54.429 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-29T22:57:54.452 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-29T22:57:54.453 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-29T22:57:54.459 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-29T22:57:54.490 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-29T22:57:54.492 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-29T22:57:54.497 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-29T22:57:54.499 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-29T22:57:54.499 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-29T22:57:54.502 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-29T22:57:54.502 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-29T22:57:54.502 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-29T22:57:54.504 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-29T22:57:54.507 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-29T22:57:54.521 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-29T22:57:54.522 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-29T22:57:54.522 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-29T22:57:54.523 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-29T22:57:54.523 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-29T22:57:54.524 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-29T22:57:54.525 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-29T22:57:54.525 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-29T22:57:54.527 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-29T22:57:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-29T22:57:54.541 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-29T22:57:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-29T22:57:54.546 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-29T22:57:54.557 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-29T22:57:54.558 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-29T22:57:54.559 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-29T22:57:54.563 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-29T22:57:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-29T22:57:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-29T22:57:54.574 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-29T22:57:54.575 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-29T22:57:54.576 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-29T22:57:54.576 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-29T22:57:54.591 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-29T22:57:54.622 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-29T22:57:54.674 INFO:teuthology.orchestra.run.smithi064.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-29T22:57:54.758 INFO:teuthology.orchestra.run.smithi064.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-29T22:57:54.759 INFO:teuthology.orchestra.run.smithi064.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-29T22:57:54.760 INFO:teuthology.orchestra.run.smithi064.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-29T22:57:54.770 INFO:teuthology.orchestra.run.smithi064.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-29T22:57:54.774 INFO:teuthology.orchestra.run.smithi064.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-29T22:57:54.774 INFO:teuthology.orchestra.run.smithi064.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-29T22:57:54.776 INFO:teuthology.orchestra.run.smithi064.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-29T22:57:54.777 INFO:teuthology.orchestra.run.smithi064.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-29T22:57:54.777 INFO:teuthology.orchestra.run.smithi064.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-29T22:57:54.778 INFO:teuthology.orchestra.run.smithi064.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-29T22:57:54.778 INFO:teuthology.orchestra.run.smithi064.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-29T22:57:54.779 INFO:teuthology.orchestra.run.smithi064.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-29T22:57:54.780 INFO:teuthology.orchestra.run.smithi064.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-29T22:57:54.780 INFO:teuthology.orchestra.run.smithi064.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-29T22:57:54.780 INFO:teuthology.orchestra.run.smithi064.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-29T22:57:54.781 INFO:teuthology.orchestra.run.smithi064.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-29T22:57:54.789 INFO:teuthology.orchestra.run.smithi064.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-29T22:57:54.792 INFO:teuthology.orchestra.run.smithi064.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-29T22:57:54.798 INFO:teuthology.orchestra.run.smithi064.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-29T22:57:54.801 INFO:teuthology.orchestra.run.smithi064.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-29T22:57:54.803 INFO:teuthology.orchestra.run.smithi064.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-29T22:57:54.805 INFO:teuthology.orchestra.run.smithi064.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-29T22:57:54.806 INFO:teuthology.orchestra.run.smithi064.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-29T22:57:54.809 INFO:teuthology.orchestra.run.smithi064.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-29T22:57:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-29T22:57:54.828 INFO:teuthology.orchestra.run.smithi064.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-29T22:57:54.830 INFO:teuthology.orchestra.run.smithi064.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-29T22:57:54.866 INFO:teuthology.orchestra.run.smithi064.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-29T22:57:54.867 INFO:teuthology.orchestra.run.smithi064.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-29T22:57:54.868 INFO:teuthology.orchestra.run.smithi064.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-29T22:57:54.868 INFO:teuthology.orchestra.run.smithi064.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-29T22:57:54.868 INFO:teuthology.orchestra.run.smithi064.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-29T22:57:54.869 INFO:teuthology.orchestra.run.smithi064.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-29T22:57:54.881 INFO:teuthology.orchestra.run.smithi064.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-29T22:57:54.881 INFO:teuthology.orchestra.run.smithi064.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-29T22:57:54.881 INFO:teuthology.orchestra.run.smithi064.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-29T22:57:54.884 INFO:teuthology.orchestra.run.smithi064.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-29T22:57:54.901 INFO:teuthology.orchestra.run.smithi064.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-29T22:57:54.901 INFO:teuthology.orchestra.run.smithi064.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-29T22:57:54.902 INFO:teuthology.orchestra.run.smithi064.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-29T22:57:54.902 INFO:teuthology.orchestra.run.smithi064.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-29T22:57:54.904 INFO:teuthology.orchestra.run.smithi064.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-29T22:57:54.905 INFO:teuthology.orchestra.run.smithi064.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-29T22:57:54.973 INFO:teuthology.orchestra.run.smithi064.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-29T22:57:54.976 INFO:teuthology.orchestra.run.smithi064.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-29T22:57:54.991 INFO:teuthology.orchestra.run.smithi064.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-29T22:57:55.994 INFO:teuthology.orchestra.run.smithi064.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-29T22:57:56.121 INFO:teuthology.orchestra.run.smithi064.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-29T22:57:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-29T22:57:56.133 INFO:teuthology.orchestra.run.smithi064.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-29T22:57:56.186 INFO:teuthology.orchestra.run.smithi064.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-29T22:57:56.443 INFO:teuthology.orchestra.run.smithi064.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-29T22:57:57.487 INFO:teuthology.orchestra.run.smithi064.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-29T22:57:57.488 INFO:teuthology.orchestra.run.smithi064.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-29T22:57:57.501 INFO:teuthology.orchestra.run.smithi064.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-29T22:57:57.519 INFO:teuthology.orchestra.run.smithi064.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-29T22:57:57.520 INFO:teuthology.orchestra.run.smithi064.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-29T22:57:57.521 INFO:teuthology.orchestra.run.smithi064.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-29T22:57:57.556 INFO:teuthology.orchestra.run.smithi064.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-29T22:57:57.814 INFO:teuthology.orchestra.run.smithi064.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-29T22:57:57.814 INFO:teuthology.orchestra.run.smithi064.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-29T22:57:59.873 INFO:teuthology.orchestra.run.smithi064.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-29T22:57:59.945 INFO:teuthology.orchestra.run.smithi064.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-29T22:57:59.946 INFO:teuthology.orchestra.run.smithi064.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-29T22:58:00.431 INFO:teuthology.orchestra.run.smithi064.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-29T22:58:01.073 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 209 MB in 6s (34.0 MB/s) 2023-09-29T22:58:01.208 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-29T22:58:01.266 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-29T22:58:01.270 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-29T22:58:01.340 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T22:58:01.694 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:01.819 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T22:58:02.485 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:02.584 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T22:58:03.280 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs2. 2023-09-29T22:58:03.297 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:03.338 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:03.666 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rados. 2023-09-29T22:58:03.682 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:03.724 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:04.101 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-29T22:58:04.118 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:04.160 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:04.412 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cephfs. 2023-09-29T22:58:04.429 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:04.470 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:04.790 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-29T22:58:04.808 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:04.848 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:05.126 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-prettytable. 2023-09-29T22:58:05.144 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-29T22:58:05.176 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-29T22:58:05.403 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rbd. 2023-09-29T22:58:05.421 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:05.478 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:05.831 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-29T22:58:05.847 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-29T22:58:05.898 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T22:58:06.251 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-29T22:58:06.269 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-29T22:58:06.309 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T22:58:06.645 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-29T22:58:06.664 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-29T22:58:06.703 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T22:58:07.098 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-29T22:58:07.115 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-29T22:58:07.164 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T22:58:07.434 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-29T22:58:07.452 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-29T22:58:07.484 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T22:58:07.971 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-29T22:58:07.987 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T22:58:08.048 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:58:08.550 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-29T22:58:08.567 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T22:58:08.617 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:58:08.945 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-29T22:58:08.964 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T22:58:09.004 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:58:09.412 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-29T22:58:09.429 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-29T22:58:09.470 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T22:58:09.823 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librgw2. 2023-09-29T22:58:09.839 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:09.881 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:10.626 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rgw. 2023-09-29T22:58:10.643 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:10.684 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:11.012 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liboath0. 2023-09-29T22:58:11.029 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-29T22:58:11.070 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-29T22:58:11.364 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libradosstriper1. 2023-09-29T22:58:11.384 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:11.447 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:12.035 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-common. 2023-09-29T22:58:12.050 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:12.092 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:14.890 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-base. 2023-09-29T22:58:14.912 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:14.958 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:16.019 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cryptography. 2023-09-29T22:58:16.037 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-29T22:58:16.077 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T22:58:16.473 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-openssl. 2023-09-29T22:58:16.490 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-29T22:58:16.531 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-29T22:58:16.800 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-29T22:58:16.817 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:16.858 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:17.248 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-29T22:58:17.265 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-29T22:58:17.306 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T22:58:17.600 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-29T22:58:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-29T22:58:17.658 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-29T22:58:18.037 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-natsort. 2023-09-29T22:58:18.054 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-29T22:58:18.095 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-29T22:58:18.398 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jwt. 2023-09-29T22:58:18.417 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-29T22:58:18.448 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T22:58:18.751 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-logutils. 2023-09-29T22:58:18.769 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-29T22:58:18.809 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-29T22:58:19.112 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-29T22:58:19.130 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-29T22:58:19.170 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-29T22:58:19.490 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-mako. 2023-09-29T22:58:19.509 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-29T22:58:19.548 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T22:58:19.868 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-29T22:58:19.888 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-29T22:58:19.926 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-29T22:58:20.238 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-29T22:58:20.254 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-29T22:58:20.312 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T22:58:20.590 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webob. 2023-09-29T22:58:20.606 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-29T22:58:20.657 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-29T22:58:20.976 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-29T22:58:20.994 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-29T22:58:21.068 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T22:58:21.371 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bs4. 2023-09-29T22:58:21.389 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-29T22:58:21.429 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-29T22:58:21.774 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-waitress. 2023-09-29T22:58:21.793 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-29T22:58:21.860 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T22:58:22.193 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempita. 2023-09-29T22:58:22.210 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-29T22:58:22.252 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-29T22:58:22.554 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-paste. 2023-09-29T22:58:22.571 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-29T22:58:22.629 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T22:58:22.949 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-29T22:58:22.969 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-29T22:58:23.007 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T22:58:23.310 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-29T22:58:23.327 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-29T22:58:23.393 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-29T22:58:23.680 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webtest. 2023-09-29T22:58:23.697 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-29T22:58:23.738 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-29T22:58:24.032 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pecan. 2023-09-29T22:58:24.052 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-29T22:58:24.099 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T22:58:24.479 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-29T22:58:24.497 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-29T22:58:24.537 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T22:58:24.874 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-29T22:58:24.890 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:24.948 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:25.302 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr. 2023-09-29T22:58:25.321 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:25.359 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:25.871 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mon. 2023-09-29T22:58:25.887 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:25.929 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:27.767 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-osd. 2023-09-29T22:58:27.783 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:27.825 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:30.669 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph. 2023-09-29T22:58:30.686 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:30.727 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:31.063 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-fuse. 2023-09-29T22:58:31.084 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:31.121 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:31.558 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mds. 2023-09-29T22:58:31.575 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:31.616 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:32.169 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package cephadm. 2023-09-29T22:58:32.186 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:32.227 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:32.547 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jinja2. 2023-09-29T22:58:32.565 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-29T22:58:32.605 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-29T22:58:33.025 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-29T22:58:33.042 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:33.428 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-29T22:58:33.444 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-29T22:58:33.485 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-29T22:58:33.730 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-underscore. 2023-09-29T22:58:33.746 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-29T22:58:33.788 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T22:58:34.057 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-29T22:58:34.076 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-29T22:58:34.115 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T22:58:34.451 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-routes. 2023-09-29T22:58:34.469 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-29T22:58:34.509 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-29T22:58:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-29T22:58:34.832 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:34.870 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:35.366 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-decorator. 2023-09-29T22:58:35.382 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-29T22:58:35.423 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T22:58:35.710 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-29T22:58:35.726 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-29T22:58:35.768 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T22:58:36.096 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-scipy. 2023-09-29T22:58:36.113 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-29T22:58:36.154 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-29T22:58:37.448 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-29T22:58:37.465 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-29T22:58:37.531 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T22:58:38.061 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-joblib. 2023-09-29T22:58:38.079 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-29T22:58:38.117 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-29T22:58:38.496 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn. 2023-09-29T22:58:38.514 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-29T22:58:38.546 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T22:58:39.238 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-29T22:58:39.255 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:39.295 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:40.390 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cachetools. 2023-09-29T22:58:40.409 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-29T22:58:40.448 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-29T22:58:40.743 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-29T22:58:40.760 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-29T22:58:40.801 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T22:58:41.113 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-29T22:58:41.129 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-29T22:58:41.170 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T22:58:41.532 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rsa. 2023-09-29T22:58:41.548 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-29T22:58:41.590 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-29T22:58:41.885 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-google-auth. 2023-09-29T22:58:41.904 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-29T22:58:41.942 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-29T22:58:42.288 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-blinker. 2023-09-29T22:58:42.304 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-29T22:58:42.346 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T22:58:42.657 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-29T22:58:42.674 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-29T22:58:42.715 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T22:58:43.361 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-29T22:58:43.377 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-29T22:58:43.418 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T22:58:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-websocket. 2023-09-29T22:58:43.746 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-29T22:58:43.788 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T22:58:44.107 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-29T22:58:44.127 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-29T22:58:44.165 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T22:58:44.679 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-29T22:58:44.697 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:44.737 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:45.133 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-29T22:58:45.150 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-29T22:58:45.191 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-29T22:58:45.537 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-29T22:58:45.553 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T22:58:45.586 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T22:58:45.847 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package jq. 2023-09-29T22:58:45.864 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T22:58:45.906 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T22:58:46.158 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package socat. 2023-09-29T22:58:46.175 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-29T22:58:46.217 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-29T22:58:46.529 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package xmlstarlet. 2023-09-29T22:58:46.548 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-29T22:58:46.587 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-29T22:58:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-test. 2023-09-29T22:58:46.950 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:46.991 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:51.954 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-volume. 2023-09-29T22:58:51.971 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-29T22:58:52.012 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:52.308 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-29T22:58:52.326 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T22:58:52.366 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T22:58:52.862 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-29T22:58:52.879 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:58:52.920 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:58:53.323 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package nvme-cli. 2023-09-29T22:58:53.340 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-29T22:58:53.399 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T22:58:53.700 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-29T22:58:53.720 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T22:58:53.767 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T22:58:55.449 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-29T22:58:55.468 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-29T22:58:55.507 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-29T22:58:55.877 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-29T22:58:55.894 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-29T22:58:55.935 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T22:58:56.213 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-attr. 2023-09-29T22:58:56.233 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-29T22:58:56.272 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-29T22:58:56.616 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webencodings. 2023-09-29T22:58:56.634 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-29T22:58:56.674 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T22:58:56.960 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-html5lib. 2023-09-29T22:58:56.981 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-29T22:58:57.018 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-29T22:58:57.388 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-29T22:58:57.407 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-29T22:58:57.446 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T22:58:57.883 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-packaging. 2023-09-29T22:58:57.902 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-29T22:58:57.966 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-29T22:58:58.252 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastescript. 2023-09-29T22:58:58.273 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-29T22:58:58.310 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-29T22:58:58.605 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pluggy. 2023-09-29T22:58:58.625 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-29T22:58:58.664 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-29T22:58:59.000 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-psutil. 2023-09-29T22:58:59.017 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-29T22:58:59.058 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T22:58:59.370 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-py. 2023-09-29T22:58:59.387 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-29T22:58:59.427 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T22:58:59.772 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-29T22:58:59.790 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-29T22:58:59.839 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T22:59:00.150 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-29T22:59:00.167 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-29T22:59:00.208 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T22:59:00.544 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pytest. 2023-09-29T22:59:00.562 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-29T22:59:00.602 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-29T22:59:00.948 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplejson. 2023-09-29T22:59:00.964 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-29T22:59:01.006 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T22:59:01.308 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-29T22:59:01.326 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-29T22:59:01.366 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T22:59:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package radosgw. 2023-09-29T22:59:02.212 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:59:02.253 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:04.016 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-fuse. 2023-09-29T22:59:04.034 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-29T22:59:04.074 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:04.394 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-29T22:59:04.697 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T22:59:04.824 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T22:59:04.968 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T22:59:05.169 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:05.788 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user cephadm....done 2023-09-29T22:59:05.941 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T22:59:06.067 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T22:59:06.414 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-29T22:59:06.698 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T22:59:06.833 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-29T22:59:06.959 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T22:59:07.269 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-29T22:59:07.688 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T22:59:07.981 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-29T22:59:08.273 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T22:59:08.575 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T22:59:08.901 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:09.185 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T22:59:09.287 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-29T22:59:09.571 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T22:59:09.881 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T22:59:10.007 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T22:59:10.383 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-29T22:59:10.713 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T22:59:10.997 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T22:59:11.332 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T22:59:11.617 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-29T22:59:11.960 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-29T22:59:12.286 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-29T22:59:12.588 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T22:59:12.712 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T22:59:12.864 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T22:59:13.150 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T22:59:13.560 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T22:59:14.287 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T22:59:14.763 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-29T22:59:14.889 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:15.232 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:59:15.350 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-29T22:59:15.693 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T22:59:15.819 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-29T22:59:15.945 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T22:59:16.572 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:59:16.707 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T22:59:16.858 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T22:59:17.184 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T22:59:17.485 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-29T22:59:17.811 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T22:59:17.937 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-29T22:59:18.230 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-29T22:59:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T22:59:18.832 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T22:59:19.256 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-29T22:59:19.527 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-29T22:59:19.655 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T22:59:19.998 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-29T22:59:20.350 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T22:59:20.459 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T22:59:20.585 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-29T22:59:20.937 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T22:59:21.238 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:21.381 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-29T22:59:22.906 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:23.056 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T22:59:23.182 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-29T22:59:23.561 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-29T22:59:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-29T22:59:24.164 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T22:59:24.289 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-29T22:59:24.649 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:24.775 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T22:59:25.094 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:25.219 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T22:59:25.554 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T22:59:26.626 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:26.752 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T22:59:26.903 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-29T22:59:27.029 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T22:59:27.155 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-29T22:59:27.465 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T22:59:27.591 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-29T22:59:27.909 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:28.018 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:28.228 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T22:59:28.228 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T22:59:28.686 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:28.881 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T22:59:29.182 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-29T22:59:29.542 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T22:59:29.903 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:30.037 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:30.163 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:30.289 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:30.415 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-29T22:59:30.725 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-29T22:59:31.052 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:31.178 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:31.295 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T22:59:32.240 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:32.793 INFO:teuthology.orchestra.run.smithi064.stdout:Adding group ceph....done 2023-09-29T22:59:33.170 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user ceph....done 2023-09-29T22:59:33.345 INFO:teuthology.orchestra.run.smithi064.stdout:Setting system user ceph properties....done 2023-09-29T22:59:33.370 INFO:teuthology.orchestra.run.smithi064.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-29T22:59:33.519 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-29T22:59:33.850 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-29T22:59:34.308 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:34.443 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:34.714 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:59:34.714 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:59:35.361 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:35.769 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-29T22:59:36.205 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:36.463 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T22:59:36.464 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T22:59:36.932 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:37.166 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T22:59:37.166 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T22:59:37.608 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:38.088 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T22:59:38.088 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T22:59:38.568 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:38.719 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:38.853 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:39.072 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T22:59:39.072 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T22:59:39.513 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:39.656 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:39.782 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:39.924 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-29T22:59:40.292 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-29T22:59:40.440 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-29T22:59:40.750 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-29T22:59:43.728 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-29T22:59:44.300 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-29T22:59:44.302 DEBUG:teuthology.parallel:result is None 2023-09-29T22:59:44.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T22:59:44.819 DEBUG:teuthology.orchestra.run.smithi064:> dpkg-query -W -f '${Version}' ceph 2023-09-29T22:59:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-29T22:59:44.844 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-29T22:59:44.844 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-29T22:59:44.844 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-29T22:59:44.845 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:44.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T22:59:44.900 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-29T22:59:44.900 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:44.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T22:59:44.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T22:59:44.932 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-29T22:59:44.932 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:44.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T22:59:44.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T22:59:45.002 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-29T22:59:45.002 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T22:59:45.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T22:59:45.025 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T22:59:45.090 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-29T22:59:45.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2023-09-29T22:59:45.105 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-29T22:59:45.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T22:59:45.116 INFO:tasks.ceph:Creating extra log directories... 2023-09-29T22:59:45.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T22:59:45.137 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-29T22:59:45.137 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T22:59:45.137 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406505', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool', 'mon/pool_ops.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T22:59:45.138 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-29T22:59:45.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T22:59:45.158 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.159 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T22:59:45.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T22:59:45.209 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-09-29 22:56:12.740459988 +0000 2023-09-29T22:59:45.216 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-09-29 22:56:12.456466534 +0000 2023-09-29T22:59:45.217 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-09-29 22:56:12.456466534 +0000 2023-09-29T22:59:45.217 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-09-29T22:59:45.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T22:59:45.229 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-09-29T22:59:45.230 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-09-29T22:59:45.230 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000402638 s, 1.3 MB/s 2023-09-29T22:59:45.231 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T22:59:45.239 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2023-09-29T22:59:45.245 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T22:59:45.245 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:59:45.245 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 368 Links: 1 2023-09-29T22:59:45.246 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:59:45.246 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-09-29 22:56:13.240448463 +0000 2023-09-29T22:59:45.246 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-09-29 22:56:12.948455194 +0000 2023-09-29T22:59:45.246 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-09-29 22:56:12.948455194 +0000 2023-09-29T22:59:45.246 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-09-29T22:59:45.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T22:59:45.257 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-09-29T22:59:45.257 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-09-29T22:59:45.257 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000406625 s, 1.3 MB/s 2023-09-29T22:59:45.258 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T22:59:45.266 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2023-09-29T22:59:45.317 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T22:59:45.317 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-09-29 22:56:13.732437124 +0000 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-09-29 22:56:13.488442747 +0000 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-09-29 22:56:13.488442747 +0000 2023-09-29T22:59:45.318 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-09-29T22:59:45.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T22:59:45.331 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-09-29T22:59:45.331 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-09-29T22:59:45.332 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000392752 s, 1.3 MB/s 2023-09-29T22:59:45.332 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T22:59:45.340 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-09-29 22:56:14.232425599 +0000 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-09-29 22:56:13.964431775 +0000 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-09-29 22:56:13.964431775 +0000 2023-09-29T22:59:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-09-29T22:59:45.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T22:59:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-09-29T22:59:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-09-29T22:59:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000412324 s, 1.2 MB/s 2023-09-29T22:59:45.402 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T22:59:45.409 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:59:45.409 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:59:45.409 INFO:tasks.ceph:Generating config... 2023-09-29T22:59:45.411 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-29T22:59:45.411 INFO:tasks.ceph:[global] ms type = async 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T22:59:45.411 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-29T22:59:45.411 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-29T22:59:45.412 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-29T22:59:45.413 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-29T22:59:45.413 INFO:tasks.ceph:Setting up mon.a... 2023-09-29T22:59:45.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-29T22:59:45.443 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2023-09-29T22:59:45.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-29T22:59:45.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T22:59:45.584 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64'), ('mon.b', '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]'), ('mon.c', '[v2:172.21.15.64:3302,v1:172.21.15.64:6791]')] 2023-09-29T22:59:45.584 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],[v2:172.21.15.64:3302,v1:172.21.15.64:6791]', 'mon election default strategy': 3, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-09-29T22:59:45.585 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.585 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:59:45.590 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --addv b '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]' --addv c '[v2:172.21.15.64:3302,v1:172.21.15.64:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:45.671 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:45.671 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid fe756412-e9c8-4096-b5b3-42c87be387e9 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = octopus 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:fsid fe756412-e9c8-4096-b5b3-42c87be387e9 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2023-09-29T22:59:45.672091+0000 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:created 2023-09-29T22:59:45.672091+0000 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 15 (octopus) 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 3 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:1: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.b 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:2: [v2:172.21.15.64:3302/0,v1:172.21.15.64:6791/0] mon.c 2023-09-29T22:59:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-29T22:59:45.674 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:59:45.678 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fe756412-e9c8-4096-b5b3-42c87be387e9... 2023-09-29T22:59:45.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-29T22:59:45.714 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.715 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.716 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2023-09-29T22:59:45.717 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],[v2:172.21.15.64:3302,v1:172.21.15.64:6791]" 2023-09-29T22:59:45.718 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 3 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = fe756412-e9c8-4096-b5b3-42c87be387e9 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2023-09-29T22:59:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = snappy 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2023-09-29T22:59:45.721 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2023-09-29T22:59:45.722 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:59:45.723 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2023-09-29T22:59:45.724 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2023-09-29T22:59:45.726 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2023-09-29T22:59:45.727 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-29T22:59:45.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-29T22:59:45.821 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T22:59:45.821 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.821 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-29T22:59:45.828 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.828 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-29T22:59:45.833 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T22:59:45.833 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-29T22:59:45.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T22:59:45.854 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:45.854 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:45.901 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T22:59:45.901 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T22:59:45.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T22:59:45.946 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T22:59:45.949 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T22:59:45.949 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T22:59:45.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-29T22:59:45.986 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-29T22:59:45.998 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T22:59:45.998 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:59:45.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-29T22:59:46.009 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:59:46.009 INFO:tasks.ceph:role: osd.0 2023-09-29T22:59:46.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T22:59:46.009 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T22:59:46.591 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:59:46.591 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:59:46.591 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:59:46.591 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:59:46.592 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:59:46.593 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-09-29T22:59:46.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-29T22:59:46.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-29T22:59:46.708 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T22:59:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:59:46.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-29T22:59:46.719 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:59:46.719 INFO:tasks.ceph:role: osd.1 2023-09-29T22:59:46.719 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T22:59:46.719 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T22:59:47.264 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:59:47.265 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:59:47.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-09-29T22:59:47.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-29T22:59:47.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-29T22:59:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:59:47.309 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T22:59:47.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-29T22:59:47.366 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:59:47.366 INFO:tasks.ceph:role: osd.2 2023-09-29T22:59:47.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T22:59:47.366 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T22:59:47.838 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:59:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:59:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:59:47.839 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:59:47.840 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-09-29T22:59:47.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-29T22:59:47.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-29T22:59:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:59:47.912 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T22:59:47.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:47.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.989+0000 7f3f8abbf3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-29T22:59:47.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.989+0000 7f3f8abbf3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-29T22:59:47.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.989+0000 7f3f8abbf3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T22:59:47.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.989+0000 7f3f8abbf3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T22:59:47.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.989+0000 7f3f8abbf3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T22:59:47.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:47.993+0000 7f3f8abbf3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-29T22:59:48.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:48.561+0000 7f3f8abbf3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:49.340+0000 7f3f8abbf3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:50.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:50.120+0000 7f3f8abbf3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:50.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-29T22:59:51.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:51.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-29T22:59:51.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-29T22:59:51.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T22:59:51.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T22:59:51.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T22:59:51.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.084+0000 7f37569df3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-29T22:59:51.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:51.652+0000 7f37569df3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:52.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:52.436+0000 7f37569df3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:53.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:53.232+0000 7f37569df3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:54.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-29T22:59:54.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:54.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-29T22:59:54.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-29T22:59:54.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T22:59:54.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T22:59:54.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T22:59:54.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.160+0000 7f63d71dd3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-29T22:59:54.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:54.724+0000 7f63d71dd3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:55.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:55.508+0000 7f63d71dd3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:56.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-09-29T22:59:56.292+0000 7f63d71dd3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T22:59:57.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-29T22:59:57.162 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T22:59:57.163 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:57.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-29T22:59:57.173 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:57.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-29T22:59:57.230 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:57.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-29T22:59:57.240 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:57.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-29T22:59:57.256 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T22:59:57.256 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-29T22:59:57.262 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T22:59:57.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDhVhdlmC55OBAAmgUk2SzzfX3iCZHtuXUkvA== 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDjVhdlbZJKOxAAELuqCzkyKiE3bQgNZPWMpw== 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2023-09-29T22:59:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDnVhdlMWVfBRAAvLyOJVnvMqoO4vV5INJSWw== 2023-09-29T22:59:57.321 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2023-09-29T22:59:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDqVhdlAG7FCRAAD9ArFXF4SpwmchyMjy18cg== 2023-09-29T22:59:57.321 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2023-09-29T22:59:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDhVhdlv3TZOhAAikqlQkWcGKdrSoqQpSLDmQ== 2023-09-29T22:59:57.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T22:59:57.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:59:57.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:59:57.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:59:57.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T22:59:57.581 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T22:59:57.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-29T22:59:57.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T22:59:57.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-29T22:59:57.707 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-29T22:59:57.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T22:59:57.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-29T22:59:57.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-29T22:59:57.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T22:59:57.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-29T22:59:57.971 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-29T22:59:58.020 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-29T22:59:58.021 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-29T22:59:58.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-29T22:59:58.023 INFO:tasks.ceph.mon.a:Started 2023-09-29T22:59:58.023 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-29T22:59:58.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-29T22:59:58.025 INFO:tasks.ceph.mon.b:Started 2023-09-29T22:59:58.025 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-29T22:59:58.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-29T22:59:58.027 INFO:tasks.ceph.mon.c:Started 2023-09-29T22:59:58.028 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-29T22:59:58.028 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-29T22:59:58.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-29T22:59:58.030 INFO:tasks.ceph.mgr.x:Started 2023-09-29T22:59:58.030 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T22:59:58.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2023-09-29T23:00:04.498 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2023-09-29T23:00:04.509 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-29T23:00:04.510 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T23:00:04.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-29T23:00:04.523 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T23:00:04.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-29T23:00:04.574 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:04.568+0000 7fae47ef6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T23:00:04.581 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-09-29T23:00:04.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-29T23:00:04.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 79e4347b-5d52-4e4e-ac60-47a88aea42a8 0 2023-09-29T23:00:04.831 INFO:teuthology.orchestra.run.smithi064.stdout:0 2023-09-29T23:00:04.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new fd06f30b-c73f-408e-b27d-4171c3dd2813 1 2023-09-29T23:00:05.050 INFO:teuthology.orchestra.run.smithi064.stdout:1 2023-09-29T23:00:05.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 4ec9ba46-45d7-4c06-9f41-5079d254c308 2 2023-09-29T23:00:05.213 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.208+0000 7fae47ef6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T23:00:05.263 INFO:teuthology.orchestra.run.smithi064.stdout:2 2023-09-29T23:00:05.271 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.268+0000 7fae47ef6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T23:00:05.276 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-29T23:00:05.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-29T23:00:05.278 INFO:tasks.ceph.osd.0:Started 2023-09-29T23:00:05.278 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-29T23:00:05.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-29T23:00:05.280 INFO:tasks.ceph.osd.1:Started 2023-09-29T23:00:05.281 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-29T23:00:05.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-29T23:00:05.282 INFO:tasks.ceph.osd.2:Started 2023-09-29T23:00:05.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:00:05.411 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.408+0000 7fae47ef6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:00:05.483 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:05.483 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:05.258148+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:00:05.499 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-29T23:00:05.499 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T23:00:05.655 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.652+0000 7fae47ef6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T23:00:05.660 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-09-29T23:00:05.656+0000 7fcc8d36c3c0 -1 Falling back to public interface 2023-09-29T23:00:05.672 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-09-29T23:00:05.668+0000 7f8931fba3c0 -1 Falling back to public interface 2023-09-29T23:00:05.709 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.704+0000 7fae47ef6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T23:00:05.801 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:00:05.904 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-09-29T23:00:05.900+0000 7f04353153c0 -1 Falling back to public interface 2023-09-29T23:00:05.906 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.900+0000 7fae47ef6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T23:00:05.972 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:05.968+0000 7fae47ef6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T23:00:05.973 INFO:teuthology.misc.health.smithi064.stdout: 2023-09-29T23:00:05.974 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":5,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:05.258148+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:00:05.985 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T23:00:06.039 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:06.036+0000 7fae47ef6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T23:00:06.471 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:06.468+0000 7fae47ef6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T23:00:06.596 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:06.592+0000 7fae47ef6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T23:00:06.927 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:06.924+0000 7fae47ef6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T23:00:07.105 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.100+0000 7fae47ef6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:00:07.172 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.168+0000 7fae47ef6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T23:00:07.226 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.220+0000 7fae47ef6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T23:00:07.285 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.280+0000 7fae47ef6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T23:00:07.565 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.560+0000 7fae47ef6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T23:00:07.624 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.620+0000 7fae47ef6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T23:00:07.692 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.688+0000 7fae47ef6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T23:00:07.859 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.856+0000 7fae47ef6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T23:00:07.927 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:07.924+0000 7fae47ef6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T23:00:08.664 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:08.660+0000 7fae47ef6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T23:00:08.712 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:08.708+0000 7fae257d7700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:00:08.712 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:08.708+0000 7fae257d7700 -1 pg_autoscaler.serve: 2023-09-29T23:00:08.713 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:08.708+0000 7fae257d7700 -1 Traceback (most recent call last): 2023-09-29T23:00:08.713 INFO:tasks.ceph.mgr.x.smithi064.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-29T23:00:08.713 INFO:tasks.ceph.mgr.x.smithi064.stderr: self._maybe_adjust() 2023-09-29T23:00:08.714 INFO:tasks.ceph.mgr.x.smithi064.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-29T23:00:08.714 INFO:tasks.ceph.mgr.x.smithi064.stderr: if self.noautoscale: 2023-09-29T23:00:08.714 INFO:tasks.ceph.mgr.x.smithi064.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:00:08.714 INFO:tasks.ceph.mgr.x.smithi064.stderr: 2023-09-29T23:00:09.537 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-09-29T23:00:09.532+0000 7f8931fba3c0 -1 osd.0 0 log_to_monitors true 2023-09-29T23:00:09.553 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-09-29T23:00:09.548+0000 7fcc8d36c3c0 -1 osd.1 0 log_to_monitors true 2023-09-29T23:00:09.711 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:09.708+0000 7fae30fee700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-29T23:00:09.711 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-09-29T23:00:09.708+0000 7fae30fee700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-29T23:00:09.756 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-09-29T23:00:09.752+0000 7f04353153c0 -1 osd.2 0 log_to_monitors true 2023-09-29T23:00:11.133 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-09-29T23:00:11.128+0000 7fcc88b0c700 -1 osd.1 0 waiting for initial osdmap 2023-09-29T23:00:11.139 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-09-29T23:00:11.136+0000 7f0430ab5700 -1 osd.2 0 waiting for initial osdmap 2023-09-29T23:00:11.140 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-09-29T23:00:11.136+0000 7f892ef7f700 -1 osd.0 0 waiting for initial osdmap 2023-09-29T23:00:11.146 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-09-29T23:00:11.140+0000 7fcc858b8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:00:11.149 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-09-29T23:00:11.144+0000 7f042d861700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:00:11.152 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-09-29T23:00:11.148+0000 7f892a506700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:00:12.287 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:00:12.571 INFO:teuthology.misc.health.smithi064.stdout: 2023-09-29T23:00:12.571 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":8,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:12.143490+0000","last_up_change":"2023-09-29T23:00:12.143490+0000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":30171},{"type":"v1","addr":"172.21.15.64:6809","nonce":30171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":30171},{"type":"v1","addr":"172.21.15.64:6811","nonce":30171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":30171},{"type":"v1","addr":"172.21.15.64:6815","nonce":30171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":30171},{"type":"v1","addr":"172.21.15.64:6813","nonce":30171}]},"public_addr":"172.21.15.64:6809/30171","cluster_addr":"172.21.15.64:6811/30171","heartbeat_back_addr":"172.21.15.64:6815/30171","heartbeat_front_addr":"172.21.15.64:6813/30171","state":["exists","up"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":30170},{"type":"v1","addr":"172.21.15.64:6801","nonce":30170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":30170},{"type":"v1","addr":"172.21.15.64:6803","nonce":30170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":30170},{"type":"v1","addr":"172.21.15.64:6807","nonce":30170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":30170},{"type":"v1","addr":"172.21.15.64:6805","nonce":30170}]},"public_addr":"172.21.15.64:6801/30170","cluster_addr":"172.21.15.64:6803/30170","heartbeat_back_addr":"172.21.15.64:6807/30170","heartbeat_front_addr":"172.21.15.64:6805/30170","state":["exists","up"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":30172},{"type":"v1","addr":"172.21.15.64:6817","nonce":30172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":30172},{"type":"v1","addr":"172.21.15.64:6819","nonce":30172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":30172},{"type":"v1","addr":"172.21.15.64:6823","nonce":30172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":30172},{"type":"v1","addr":"172.21.15.64:6821","nonce":30172}]},"public_addr":"172.21.15.64:6817/30172","cluster_addr":"172.21.15.64:6819/30172","heartbeat_back_addr":"172.21.15.64:6823/30172","heartbeat_front_addr":"172.21.15.64:6821/30172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:00:12.583 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-29T23:00:12.583 INFO:tasks.ceph:Creating RBD pool 2023-09-29T23:00:12.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-29T23:00:13.146 INFO:tasks.ceph.mon.a.smithi064.stderr:2023-09-29T23:00:13.140+0000 7fe21c8f8700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-29T23:00:13.159 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2023-09-29T23:00:13.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T23:00:14.166 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T23:00:14.178 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-29T23:00:14.178 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-29T23:00:14.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-29T23:00:14.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-29T23:00:14.179 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T23:00:14.514 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:14.527 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":29930},{"type":"v1","addr":"172.21.15.64:6825","nonce":29930}]},"active_addr":"172.21.15.64:6825/29930","active_change":"2023-09-29T23:00:08.684842+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":801824365}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":749140799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":482476192}]}]} 2023-09-29T23:00:14.529 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-29T23:00:14.529 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-29T23:00:14.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:00:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:14.820 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:14.147650+0000","last_up_change":"2023-09-29T23:00:12.143490+0000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:00:12.747815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:00:12.909151+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":30171},{"type":"v1","addr":"172.21.15.64:6809","nonce":30171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":30171},{"type":"v1","addr":"172.21.15.64:6811","nonce":30171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":30171},{"type":"v1","addr":"172.21.15.64:6815","nonce":30171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":30171},{"type":"v1","addr":"172.21.15.64:6813","nonce":30171}]},"public_addr":"172.21.15.64:6809/30171","cluster_addr":"172.21.15.64:6811/30171","heartbeat_back_addr":"172.21.15.64:6815/30171","heartbeat_front_addr":"172.21.15.64:6813/30171","state":["exists","up"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":30170},{"type":"v1","addr":"172.21.15.64:6801","nonce":30170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":30170},{"type":"v1","addr":"172.21.15.64:6803","nonce":30170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":30170},{"type":"v1","addr":"172.21.15.64:6807","nonce":30170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":30170},{"type":"v1","addr":"172.21.15.64:6805","nonce":30170}]},"public_addr":"172.21.15.64:6801/30170","cluster_addr":"172.21.15.64:6803/30170","heartbeat_back_addr":"172.21.15.64:6807/30170","heartbeat_front_addr":"172.21.15.64:6805/30170","state":["exists","up"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":30172},{"type":"v1","addr":"172.21.15.64:6817","nonce":30172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":30172},{"type":"v1","addr":"172.21.15.64:6819","nonce":30172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":30172},{"type":"v1","addr":"172.21.15.64:6823","nonce":30172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":30172},{"type":"v1","addr":"172.21.15.64:6821","nonce":30172}]},"public_addr":"172.21.15.64:6817/30172","cluster_addr":"172.21.15.64:6819/30172","heartbeat_back_addr":"172.21.15.64:6823/30172","heartbeat_front_addr":"172.21.15.64:6821/30172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.522113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.605074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.723856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:00:14.834 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-29T23:00:14.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:00:15.113 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:15.113 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:14.147650+0000","last_up_change":"2023-09-29T23:00:12.143490+0000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:00:12.747815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:00:12.909151+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":30171},{"type":"v1","addr":"172.21.15.64:6809","nonce":30171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":30171},{"type":"v1","addr":"172.21.15.64:6811","nonce":30171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":30171},{"type":"v1","addr":"172.21.15.64:6815","nonce":30171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":30171},{"type":"v1","addr":"172.21.15.64:6813","nonce":30171}]},"public_addr":"172.21.15.64:6809/30171","cluster_addr":"172.21.15.64:6811/30171","heartbeat_back_addr":"172.21.15.64:6815/30171","heartbeat_front_addr":"172.21.15.64:6813/30171","state":["exists","up"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":30170},{"type":"v1","addr":"172.21.15.64:6801","nonce":30170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":30170},{"type":"v1","addr":"172.21.15.64:6803","nonce":30170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":30170},{"type":"v1","addr":"172.21.15.64:6807","nonce":30170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":30170},{"type":"v1","addr":"172.21.15.64:6805","nonce":30170}]},"public_addr":"172.21.15.64:6801/30170","cluster_addr":"172.21.15.64:6803/30170","heartbeat_back_addr":"172.21.15.64:6807/30170","heartbeat_front_addr":"172.21.15.64:6805/30170","state":["exists","up"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":30172},{"type":"v1","addr":"172.21.15.64:6817","nonce":30172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":30172},{"type":"v1","addr":"172.21.15.64:6819","nonce":30172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":30172},{"type":"v1","addr":"172.21.15.64:6823","nonce":30172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":30172},{"type":"v1","addr":"172.21.15.64:6821","nonce":30172}]},"public_addr":"172.21.15.64:6817/30172","cluster_addr":"172.21.15.64:6819/30172","heartbeat_back_addr":"172.21.15.64:6823/30172","heartbeat_front_addr":"172.21.15.64:6821/30172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.522113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.605074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.723856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:00:15.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-29T23:00:15.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-29T23:00:15.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-29T23:00:15.274 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:15.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:00:15.294 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:15.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:00:15.299 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2023-09-29T23:00:15.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:00:15.568 INFO:teuthology.orchestra.run.smithi064.stdout:34359738370 2023-09-29T23:00:15.583 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:15.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-09-29T23:00:15.586 INFO:teuthology.orchestra.run.smithi064.stdout:34359738370 2023-09-29T23:00:15.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-09-29T23:00:15.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-09-29T23:00:16.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:00:16.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:00:16.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:00:16.882 INFO:teuthology.orchestra.run.smithi064.stdout:34359738370 2023-09-29T23:00:16.886 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:16.891 INFO:teuthology.orchestra.run.smithi064.stdout:34359738370 2023-09-29T23:00:16.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-09-29T23:00:16.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-09-29T23:00:16.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-09-29T23:00:17.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:00:17.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:00:17.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:00:18.196 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2023-09-29T23:00:18.213 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-09-29T23:00:18.213 DEBUG:teuthology.parallel:result is None 2023-09-29T23:00:18.219 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:18.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.0 2023-09-29T23:00:18.233 DEBUG:teuthology.parallel:result is None 2023-09-29T23:00:18.254 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2023-09-29T23:00:18.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2023-09-29T23:00:18.271 DEBUG:teuthology.parallel:result is None 2023-09-29T23:00:18.271 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-29T23:00:18.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:00:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:18.557 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:00:18.568 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-09-29T23:00:16.695598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15840,"kb_used_data":360,"kb_used_omap":0,"kb_used_meta":15424,"kb_avail":283099680,"statfs":{"total":289910292480,"available":289894072320,"internally_reserved":0,"allocated":368640,"data_stored":878400,"data_compressed":42846,"data_compressed_allocated":57344,"data_compressed_original":778240,"omap_allocated":0,"internal_metadata":15794176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366816,"statfs":{"total":96636764160,"available":96631619584,"internally_reserved":0,"allocated":90112,"data_stored":22456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5408,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366432,"statfs":{"total":96636764160,"available":96631226368,"internally_reserved":0,"allocated":139264,"data_stored":427972,"data_compressed":21423,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5408,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366432,"statfs":{"total":96636764160,"available":96631226368,"internally_reserved":0,"allocated":139264,"data_stored":427972,"data_compressed":21423,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:00:18.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:00:18.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:18.846 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:00:18.855 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-09-29T23:00:18.696183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15552,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15925248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000585"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366808,"statfs":{"total":96636764160,"available":96631611392,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:00:18.856 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-29T23:00:18.856 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-29T23:00:18.856 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-29T23:00:18.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:19.172 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:22.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:22.520 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:22.520 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:25.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:25.873 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:28.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:29.220 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:29.220 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:32.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:32.560 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:32.560 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:35.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:35.912 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:35.912 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:38.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:39.252 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:42.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:42.594 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:42.594 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:45.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:45.933 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:45.933 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:48.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:49.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:49.273 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:52.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:52.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:52.608 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:55.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:55.953 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:55.953 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:00:58.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:00:59.292 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:00:59.292 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:02.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:02.635 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:05.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:05.976 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:08.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:09.330 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:09.330 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:12.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:12.668 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:12.668 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:15.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:16.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:16.013 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:19.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:19.358 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:22.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:22.709 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:25.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:26.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:26.049 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:29.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:29.397 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:29.397 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:32.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:32.740 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:32.740 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:35.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:36.080 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:39.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:39.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:39.427 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:42.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:42.762 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:42.762 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:45.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:46.108 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:49.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:49.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:49.457 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:52.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:52.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:52.796 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:55.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:01:59.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:01:59.480 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:01:59.480 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:02.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:02.827 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:05.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:06.170 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:09.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:09.513 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:09.513 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:12.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:12.854 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:12.854 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:15.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:16.195 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:19.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:19.544 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:22.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:22.889 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:22.889 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:25.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:26.231 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:26.232 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:29.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:29.566 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:32.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:32.906 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:35.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:36.244 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:39.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:39.584 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:39.584 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:42.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:42.926 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:42.926 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:45.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:46.266 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:46.266 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:49.281 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:52.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:52.945 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:52.945 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:55.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:56.280 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:56.281 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:02:59.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:02:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:02:59.613 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:02.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:02.955 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:05.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:06.303 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:09.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:09.652 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:09.653 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:12.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:12.995 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:16.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:16.330 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:19.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:19.681 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:22.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:23.028 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:23.029 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:26.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:26.378 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:26.378 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:29.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:29.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:29.718 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:32.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:33.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:33.054 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:36.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:36.400 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:36.400 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:39.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:39.749 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:39.750 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:42.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:43.091 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:46.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:46.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:46.426 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:49.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:49.761 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:52.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:53.105 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:53.105 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:56.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:56.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:56.447 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:03:59.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:03:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:03:59.784 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:02.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:03.121 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:03.122 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:06.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:06.462 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:06.463 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:09.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:09.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:09.799 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:12.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:13.145 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:16.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:16.492 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:16.492 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:19.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:19.832 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:22.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:23.177 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:23.177 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:26.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:26.521 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:29.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:29.858 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:29.859 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:32.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:33.205 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:33.205 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:36.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:36.544 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:36.545 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:39.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:39.892 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:42.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:43.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:43.232 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:46.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:46.566 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:49.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:49.919 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:49.919 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:52.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:53.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:53.257 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:56.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:56.598 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:56.599 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:04:59.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:04:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:04:59.944 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:02.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:03.291 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:06.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:06.628 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:09.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:09.968 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:09.969 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:12.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:13.309 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:16.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:16.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:16.659 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:19.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:05:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:19.996 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:05:20.009 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-29T23:05:20.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:05:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:20.301 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":11,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:15.150837+0000","last_up_change":"2023-09-29T23:00:12.143490+0000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:00:12.747815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:00:12.909151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":30171},{"type":"v1","addr":"172.21.15.64:6809","nonce":30171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":30171},{"type":"v1","addr":"172.21.15.64:6811","nonce":30171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":30171},{"type":"v1","addr":"172.21.15.64:6815","nonce":30171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":30171},{"type":"v1","addr":"172.21.15.64:6813","nonce":30171}]},"public_addr":"172.21.15.64:6809/30171","cluster_addr":"172.21.15.64:6811/30171","heartbeat_back_addr":"172.21.15.64:6815/30171","heartbeat_front_addr":"172.21.15.64:6813/30171","state":["exists","up"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":30170},{"type":"v1","addr":"172.21.15.64:6801","nonce":30170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":30170},{"type":"v1","addr":"172.21.15.64:6803","nonce":30170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":30170},{"type":"v1","addr":"172.21.15.64:6807","nonce":30170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":30170},{"type":"v1","addr":"172.21.15.64:6805","nonce":30170}]},"public_addr":"172.21.15.64:6801/30170","cluster_addr":"172.21.15.64:6803/30170","heartbeat_back_addr":"172.21.15.64:6807/30170","heartbeat_front_addr":"172.21.15.64:6805/30170","state":["exists","up"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":30172},{"type":"v1","addr":"172.21.15.64:6817","nonce":30172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":30172},{"type":"v1","addr":"172.21.15.64:6819","nonce":30172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":30172},{"type":"v1","addr":"172.21.15.64:6823","nonce":30172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":30172},{"type":"v1","addr":"172.21.15.64:6821","nonce":30172}]},"public_addr":"172.21.15.64:6817/30172","cluster_addr":"172.21.15.64:6819/30172","heartbeat_back_addr":"172.21.15.64:6823/30172","heartbeat_front_addr":"172.21.15.64:6821/30172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.522113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.605074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.723856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:05:20.315 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-29T23:05:20.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:05:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:20.602 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:05:20.612 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2023-09-29T23:05:18.780433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15552,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15925248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003787"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366808,"statfs":{"total":96636764160,"available":96631611392,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.45700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.315,"15min":0.315},"max":{"1min":0.64900000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.48999999999999999},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.48899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:05:20.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:05:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:20.902 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:05:20.914 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2023-09-29T23:05:20.780949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15552,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15925248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366808,"statfs":{"total":96636764160,"available":96631611392,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.45700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.315,"15min":0.315},"max":{"1min":0.64900000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.48999999999999999},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.48899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:05:20.915 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-29T23:05:20.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:05:21.202 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:21.205 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:05:21.216 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2023-09-29T23:05:20.780949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15552,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15925248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366808,"statfs":{"total":96636764160,"available":96631611392,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.45700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.315,"15min":0.315},"max":{"1min":0.64900000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.48999999999999999},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.48899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:05:21.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:05:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:21.502 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":11,"fsid":"fe756412-e9c8-4096-b5b3-42c87be387e9","created":"2023-09-29T23:00:04.363329+0000","modified":"2023-09-29T23:00:15.150837+0000","last_up_change":"2023-09-29T23:00:12.143490+0000","last_in_change":"2023-09-29T23:00:05.258148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:00:12.747815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:00:12.909151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79e4347b-5d52-4e4e-ac60-47a88aea42a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":30171},{"type":"v1","addr":"172.21.15.64:6809","nonce":30171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":30171},{"type":"v1","addr":"172.21.15.64:6811","nonce":30171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":30171},{"type":"v1","addr":"172.21.15.64:6815","nonce":30171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":30171},{"type":"v1","addr":"172.21.15.64:6813","nonce":30171}]},"public_addr":"172.21.15.64:6809/30171","cluster_addr":"172.21.15.64:6811/30171","heartbeat_back_addr":"172.21.15.64:6815/30171","heartbeat_front_addr":"172.21.15.64:6813/30171","state":["exists","up"]},{"osd":1,"uuid":"fd06f30b-c73f-408e-b27d-4171c3dd2813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":30170},{"type":"v1","addr":"172.21.15.64:6801","nonce":30170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":30170},{"type":"v1","addr":"172.21.15.64:6803","nonce":30170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":30170},{"type":"v1","addr":"172.21.15.64:6807","nonce":30170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":30170},{"type":"v1","addr":"172.21.15.64:6805","nonce":30170}]},"public_addr":"172.21.15.64:6801/30170","cluster_addr":"172.21.15.64:6803/30170","heartbeat_back_addr":"172.21.15.64:6807/30170","heartbeat_front_addr":"172.21.15.64:6805/30170","state":["exists","up"]},{"osd":2,"uuid":"4ec9ba46-45d7-4c06-9f41-5079d254c308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":30172},{"type":"v1","addr":"172.21.15.64:6817","nonce":30172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":30172},{"type":"v1","addr":"172.21.15.64:6819","nonce":30172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":30172},{"type":"v1","addr":"172.21.15.64:6823","nonce":30172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":30172},{"type":"v1","addr":"172.21.15.64:6821","nonce":30172}]},"public_addr":"172.21.15.64:6817/30172","cluster_addr":"172.21.15.64:6819/30172","heartbeat_back_addr":"172.21.15.64:6823/30172","heartbeat_front_addr":"172.21.15.64:6821/30172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.522113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.605074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:00:10.723856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:05:22.517 INFO:tasks.ceph:Scrubbing osd.0 2023-09-29T23:05:22.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:05:22.669 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-09-29T23:05:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:05:22.670 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-09-29T23:05:22.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-29T23:05:22.970 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2023-09-29T23:05:22.980 INFO:tasks.ceph:Scrubbing osd.1 2023-09-29T23:05:22.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:05:23.119 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-09-29T23:05:23.119 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:05:23.119 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-09-29T23:05:23.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-29T23:05:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2023-09-29T23:05:23.434 INFO:tasks.ceph:Scrubbing osd.2 2023-09-29T23:05:23.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:05:23.577 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-09-29T23:05:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:05:23.577 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-09-29T23:05:23.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-29T23:05:23.874 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2023-09-29T23:05:23.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:05:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:24.170 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:05:24.180 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-29T23:05:22.781365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15552,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15925248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003457"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163577+0000","last_change":"2023-09-29T23:00:14.161031+0000","last_active":"2023-09-29T23:00:14.163577+0000","last_peered":"2023-09-29T23:00:14.163577+0000","last_clean":"2023-09-29T23:00:14.163577+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:00:14.163577+0000","last_undegraded":"2023-09-29T23:00:14.163577+0000","last_fullsized":"2023-09-29T23:00:14.163577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:10.483106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163530+0000","last_change":"2023-09-29T23:00:14.161307+0000","last_active":"2023-09-29T23:00:14.163530+0000","last_peered":"2023-09-29T23:00:14.163530+0000","last_clean":"2023-09-29T23:00:14.163530+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:00:14.163530+0000","last_undegraded":"2023-09-29T23:00:14.163530+0000","last_fullsized":"2023-09-29T23:00:14.163530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:55.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163527+0000","last_change":"2023-09-29T23:00:14.161205+0000","last_active":"2023-09-29T23:00:14.163527+0000","last_peered":"2023-09-29T23:00:14.163527+0000","last_clean":"2023-09-29T23:00:14.163527+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:00:14.163527+0000","last_undegraded":"2023-09-29T23:00:14.163527+0000","last_fullsized":"2023-09-29T23:00:14.163527+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:11:44.455190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163400+0000","last_change":"2023-09-29T23:00:14.160275+0000","last_active":"2023-09-29T23:00:14.163400+0000","last_peered":"2023-09-29T23:00:14.163400+0000","last_clean":"2023-09-29T23:00:14.163400+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:00:14.163400+0000","last_undegraded":"2023-09-29T23:00:14.163400+0000","last_fullsized":"2023-09-29T23:00:14.163400+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:13:50.913667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163051+0000","last_change":"2023-09-29T23:00:14.161363+0000","last_active":"2023-09-29T23:00:14.163051+0000","last_peered":"2023-09-29T23:00:14.163051+0000","last_clean":"2023-09-29T23:00:14.163051+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:00:14.163051+0000","last_undegraded":"2023-09-29T23:00:14.163051+0000","last_fullsized":"2023-09-29T23:00:14.163051+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:31.755647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.161887+0000","last_change":"2023-09-29T23:00:14.160554+0000","last_active":"2023-09-29T23:00:14.161887+0000","last_peered":"2023-09-29T23:00:14.161887+0000","last_clean":"2023-09-29T23:00:14.161887+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:00:14.161887+0000","last_undegraded":"2023-09-29T23:00:14.161887+0000","last_fullsized":"2023-09-29T23:00:14.161887+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:16:23.429071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.162839+0000","last_change":"2023-09-29T23:00:14.161241+0000","last_active":"2023-09-29T23:00:14.162839+0000","last_peered":"2023-09-29T23:00:14.162839+0000","last_clean":"2023-09-29T23:00:14.162839+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:00:14.162839+0000","last_undegraded":"2023-09-29T23:00:14.162839+0000","last_fullsized":"2023-09-29T23:00:14.162839+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:10:06.137077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:14.163292+0000","last_change":"2023-09-29T23:00:14.158662+0000","last_active":"2023-09-29T23:00:14.163292+0000","last_peered":"2023-09-29T23:00:14.163292+0000","last_clean":"2023-09-29T23:00:14.163292+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:00:14.163292+0000","last_undegraded":"2023-09-29T23:00:14.163292+0000","last_fullsized":"2023-09-29T23:00:14.163292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:20.107047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T23:00:15.037871+0000","last_change":"2023-09-29T23:00:14.160939+0000","last_active":"2023-09-29T23:00:15.037871+0000","last_peered":"2023-09-29T23:00:15.037871+0000","last_clean":"2023-09-29T23:00:15.037871+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:00:15.037871+0000","last_undegraded":"2023-09-29T23:00:15.037871+0000","last_fullsized":"2023-09-29T23:00:15.037871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:00:13.145381+0000","last_clean_scrub_stamp":"2023-09-29T23:00:13.145381+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:41:01.008005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366808,"statfs":{"total":96636764160,"available":96631611392,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.56899999999999995},"min":{"1min":0.45700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.315,"15min":0.315},"max":{"1min":0.64900000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.48999999999999999},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.48899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5492,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366348,"statfs":{"total":96636764160,"available":96631140352,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69099999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:04:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85999999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.184 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.185 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.185 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.185 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-29T23:00:13.145381+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=0, tm_sec=13, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=5, tm_sec=21, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:05:24.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-29T23:05:44.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:05:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-09-29T23:05:44.489 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-09-29T23:05:44.499 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2023-09-29T23:05:42.786945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16096,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":15680,"kb_avail":283099424,"statfs":{"total":289910292480,"available":289893810176,"internally_reserved":0,"allocated":401408,"data_stored":1012788,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":16056320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:30.088515+0000","last_change":"2023-09-29T23:05:30.088443+0000","last_active":"2023-09-29T23:05:30.088515+0000","last_peered":"2023-09-29T23:05:30.088515+0000","last_clean":"2023-09-29T23:05:30.088515+0000","last_became_active":"2023-09-29T23:00:14.160444+0000","last_became_peered":"2023-09-29T23:00:14.160444+0000","last_unstale":"2023-09-29T23:05:30.088515+0000","last_undegraded":"2023-09-29T23:05:30.088515+0000","last_fullsized":"2023-09-29T23:05:30.088515+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:30.088194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:30.088194+0000","last_clean_scrub_stamp":"2023-09-29T23:05:30.088194+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:37:14.251869+0000","scrub_duration":0.0036114480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:34.041246+0000","last_change":"2023-09-29T23:05:34.041176+0000","last_active":"2023-09-29T23:05:34.041246+0000","last_peered":"2023-09-29T23:05:34.041246+0000","last_clean":"2023-09-29T23:05:34.041246+0000","last_became_active":"2023-09-29T23:00:14.160148+0000","last_became_peered":"2023-09-29T23:00:14.160148+0000","last_unstale":"2023-09-29T23:05:34.041246+0000","last_undegraded":"2023-09-29T23:05:34.041246+0000","last_fullsized":"2023-09-29T23:05:34.041246+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:34.040936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:34.040936+0000","last_clean_scrub_stamp":"2023-09-29T23:05:34.040936+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:15.729933+0000","scrub_duration":0.0036678599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:32.092883+0000","last_change":"2023-09-29T23:05:32.092801+0000","last_active":"2023-09-29T23:05:32.092883+0000","last_peered":"2023-09-29T23:05:32.092883+0000","last_clean":"2023-09-29T23:05:32.092883+0000","last_became_active":"2023-09-29T23:00:14.160055+0000","last_became_peered":"2023-09-29T23:00:14.160055+0000","last_unstale":"2023-09-29T23:05:32.092883+0000","last_undegraded":"2023-09-29T23:05:32.092883+0000","last_fullsized":"2023-09-29T23:05:32.092883+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:32.092561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:32.092561+0000","last_clean_scrub_stamp":"2023-09-29T23:05:32.092561+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:06:26.976920+0000","scrub_duration":0.0042292329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:26.140194+0000","last_change":"2023-09-29T23:05:26.140085+0000","last_active":"2023-09-29T23:05:26.140194+0000","last_peered":"2023-09-29T23:05:26.140194+0000","last_clean":"2023-09-29T23:05:26.140194+0000","last_became_active":"2023-09-29T23:00:14.159952+0000","last_became_peered":"2023-09-29T23:00:14.159952+0000","last_unstale":"2023-09-29T23:05:26.140194+0000","last_undegraded":"2023-09-29T23:05:26.140194+0000","last_fullsized":"2023-09-29T23:05:26.140194+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:26.139844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:26.139844+0000","last_clean_scrub_stamp":"2023-09-29T23:05:26.139844+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:15:32.423207+0000","scrub_duration":0.0036158100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:23.573160+0000","last_change":"2023-09-29T23:05:23.573110+0000","last_active":"2023-09-29T23:05:23.573160+0000","last_peered":"2023-09-29T23:05:23.573160+0000","last_clean":"2023-09-29T23:05:23.573160+0000","last_became_active":"2023-09-29T23:00:14.160518+0000","last_became_peered":"2023-09-29T23:00:14.160518+0000","last_unstale":"2023-09-29T23:05:23.573160+0000","last_undegraded":"2023-09-29T23:05:23.573160+0000","last_fullsized":"2023-09-29T23:05:23.573160+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:23.572884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:23.572884+0000","last_clean_scrub_stamp":"2023-09-29T23:05:23.572884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:57:43.329279+0000","scrub_duration":0.0031052079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:27.719044+0000","last_change":"2023-09-29T23:05:27.718980+0000","last_active":"2023-09-29T23:05:27.719044+0000","last_peered":"2023-09-29T23:05:27.719044+0000","last_clean":"2023-09-29T23:05:27.719044+0000","last_became_active":"2023-09-29T23:00:14.160238+0000","last_became_peered":"2023-09-29T23:00:14.160238+0000","last_unstale":"2023-09-29T23:05:27.719044+0000","last_undegraded":"2023-09-29T23:05:27.719044+0000","last_fullsized":"2023-09-29T23:05:27.719044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:27.718740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:27.718740+0000","last_clean_scrub_stamp":"2023-09-29T23:05:27.718740+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:15:34.002103+0000","scrub_duration":0.0036124640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:25.719808+0000","last_change":"2023-09-29T23:05:25.719708+0000","last_active":"2023-09-29T23:05:25.719808+0000","last_peered":"2023-09-29T23:05:25.719808+0000","last_clean":"2023-09-29T23:05:25.719808+0000","last_became_active":"2023-09-29T23:00:14.160929+0000","last_became_peered":"2023-09-29T23:00:14.160929+0000","last_unstale":"2023-09-29T23:05:25.719808+0000","last_undegraded":"2023-09-29T23:05:25.719808+0000","last_fullsized":"2023-09-29T23:05:25.719808+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:25.719322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:25.719322+0000","last_clean_scrub_stamp":"2023-09-29T23:05:25.719322+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:39.615769+0000","scrub_duration":0.0041084119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:35.007509+0000","last_change":"2023-09-29T23:05:35.007392+0000","last_active":"2023-09-29T23:05:35.007509+0000","last_peered":"2023-09-29T23:05:35.007509+0000","last_clean":"2023-09-29T23:05:35.007509+0000","last_became_active":"2023-09-29T23:00:14.158367+0000","last_became_peered":"2023-09-29T23:00:14.158367+0000","last_unstale":"2023-09-29T23:05:35.007509+0000","last_undegraded":"2023-09-29T23:05:35.007509+0000","last_fullsized":"2023-09-29T23:05:35.007509+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:05:35.006974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:05:35.006974+0000","last_clean_scrub_stamp":"2023-09-29T23:05:35.006974+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:21:32.082653+0000","scrub_duration":0.0041231799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'99","reported_seq":149,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T23:05:24.136621+0000","last_change":"2023-09-29T23:05:24.136578+0000","last_active":"2023-09-29T23:05:24.136621+0000","last_peered":"2023-09-29T23:05:24.136621+0000","last_clean":"2023-09-29T23:05:24.136621+0000","last_became_active":"2023-09-29T23:00:14.160555+0000","last_became_peered":"2023-09-29T23:00:14.160555+0000","last_unstale":"2023-09-29T23:05:24.136621+0000","last_undegraded":"2023-09-29T23:05:24.136621+0000","last_fullsized":"2023-09-29T23:05:24.136621+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'99","last_scrub_stamp":"2023-09-29T23:05:24.136419+0000","last_deep_scrub":"11'99","last_deep_scrub_stamp":"2023-09-29T23:05:24.136419+0000","last_clean_scrub_stamp":"2023-09-29T23:05:24.136419+0000","objects_scrubbed":2,"log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:38.032867+0000","scrub_duration":0.012006058999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48768,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738436,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55200000000000005,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73399999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.315,"15min":0.315},"max":{"1min":0.71199999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Fri Sep 29 23:05:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.47199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77200000000000002,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738437,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5532,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94366308,"statfs":{"total":96636764160,"available":96631099392,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.48999999999999999},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.48899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.497,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.71799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5468,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366372,"statfs":{"total":96636764160,"available":96631164928,"internally_reserved":0,"allocated":151552,"data_stored":493248,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53800000000000003,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71899999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624}]},{"osd":2,"last update":"Fri Sep 29 23:05:21 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69099999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.47999999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.68400000000000005,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24384,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:05:44.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-29T23:05:44.810 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:05:44.812 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-29T23:05:44.812 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-29T23:05:44.812 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-29T23:05:44.812 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:44.882 INFO:tasks.ceph.osd.0:Stopped 2023-09-29T23:05:44.882 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-29T23:05:44.882 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:44.957 INFO:tasks.ceph.osd.1:Stopped 2023-09-29T23:05:44.958 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-29T23:05:44.958 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:45.040 INFO:tasks.ceph.osd.2:Stopped 2023-09-29T23:05:45.041 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-29T23:05:45.041 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-29T23:05:45.041 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:45.068 INFO:tasks.ceph.mgr.x:Stopped 2023-09-29T23:05:45.068 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-29T23:05:45.069 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-29T23:05:45.069 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:45.086 INFO:tasks.ceph.mon.a:Stopped 2023-09-29T23:05:45.087 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-29T23:05:45.087 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:45.102 INFO:tasks.ceph.mon.b:Stopped 2023-09-29T23:05:45.102 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-29T23:05:45.102 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:05:45.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-29T23:05:45.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-29T23:05:45.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-29T23:05:45.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-29T23:05:45.120 INFO:tasks.ceph.mon.c:Stopped 2023-09-29T23:05:45.120 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-29T23:05:45.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:05:45.142 INFO:teuthology.orchestra.run.smithi064.stdout:1696028408.71283 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:05:45.142 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-29T23:05:45.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:05:45.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:05:45.220 INFO:teuthology.orchestra.run.smithi064.stdout:1696028408.71283 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:05:45.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T23:05:45.220 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-29T23:05:45.465 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T23:05:45.465 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-29T23:05:45.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T23:05:45.741 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-29T23:05:46.152 INFO:tasks.ceph:Archiving mon data... 2023-09-29T23:05:46.153 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/data/mon.a.tgz 2023-09-29T23:05:46.153 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-09-29T23:05:46.160 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.xPLWW8j1vM 2023-09-29T23:05:46.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.xPLWW8j1vM 2023-09-29T23:05:46.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.xPLWW8j1vM 2023-09-29T23:05:46.232 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.xPLWW8j1vM is 185KB 2023-09-29T23:05:46.250 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.xPLWW8j1vM 2023-09-29T23:05:46.256 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/data/mon.b.tgz 2023-09-29T23:05:46.256 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-09-29T23:05:46.305 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.r8Zb3WlIfd 2023-09-29T23:05:46.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.r8Zb3WlIfd 2023-09-29T23:05:46.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.r8Zb3WlIfd 2023-09-29T23:05:46.452 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.r8Zb3WlIfd is 250KB 2023-09-29T23:05:46.470 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.r8Zb3WlIfd 2023-09-29T23:05:46.476 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/data/mon.c.tgz 2023-09-29T23:05:46.476 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-09-29T23:05:46.525 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.Hd3ki555Fp 2023-09-29T23:05:46.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Hd3ki555Fp 2023-09-29T23:05:46.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.Hd3ki555Fp 2023-09-29T23:05:46.675 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.Hd3ki555Fp is 250KB 2023-09-29T23:05:46.737 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.Hd3ki555Fp 2023-09-29T23:05:46.744 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-29T23:05:46.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:05:46.759 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-29T23:05:46.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:05:46.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:05:46.785 INFO:tasks.ceph:Archiving crash dumps... 2023-09-29T23:05:46.787 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/remote/smithi064/crash 2023-09-29T23:05:46.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:05:46.844 INFO:tasks.ceph:Compressing logs... 2023-09-29T23:05:46.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:05:47.668 INFO:tasks.ceph:Archiving logs... 2023-09-29T23:05:47.668 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/remote/smithi064/log 2023-09-29T23:05:47.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:05:47.880 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:05:47.951 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5ff0210e998e4b3c8cb07c8e2967f887 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:05:47.954 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-29T23:05:47.973 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-29T23:05:48.007 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-29T23:05:48.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:05:48.063 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-29T23:05:48.071 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-29T23:05:48.072 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:05:48.100 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T23:05:48.100 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-09-29T23:05:48.100 INFO:teuthology.orchestra.run.smithi064.stdout:*hv01.front.sepi 44.4.53.2 2 u 14 64 377 0.083 0.797 0.407 2023-09-29T23:05:48.100 INFO:teuthology.orchestra.run.smithi064.stdout:+hv02.front.sepi 63.231.80.2 3 u 12 64 377 0.080 0.882 0.436 2023-09-29T23:05:48.101 INFO:teuthology.orchestra.run.smithi064.stdout:+hv03.front.sepi 74.6.168.72 3 u 15 64 377 0.082 0.538 0.844 2023-09-29T23:05:48.101 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T23:05:48.102 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-29T23:05:48.112 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-29T23:05:48.112 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-29T23:05:48.141 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-29T23:05:48.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-29T23:05:48.208 INFO:teuthology.task.internal:Duration was 825.276097 seconds 2023-09-29T23:05:48.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-29T23:05:48.234 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-29T23:05:48.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:05:48.289 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-29T23:05:48.289 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2023-09-29T23:05:48.290 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:05:48.302 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-29T23:05:48.302 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:05:48.328 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-29T23:05:48.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:05:48.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-29T23:05:48.459 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-29T23:05:48.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:05:48.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-29T23:05:48.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:05:48.516 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2023-09-29T23:05:48.532 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:05:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:05:48.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-29T23:05:48.592 INFO:teuthology.task.internal:Transferring archived files... 2023-09-29T23:05:48.593 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505/remote/smithi064 2023-09-29T23:05:48.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:05:48.615 INFO:teuthology.task.internal:Removing archive directory... 2023-09-29T23:05:48.616 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:05:48.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-29T23:05:48.633 INFO:teuthology.task.internal:Not uploading archives. 2023-09-29T23:05:48.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-29T23:05:48.650 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-29T23:05:48.651 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:05:48.670 INFO:teuthology.orchestra.run.smithi064.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:05 /home/ubuntu/cephtest 2023-09-29T23:05:48.672 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-29T23:05:48.683 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-29T23:05:48.734 INFO:teuthology.nuke:Checking targets against current locks 2023-09-29T23:05:48.760 DEBUG:teuthology.nuke:shortname: smithi064 2023-09-29T23:05:48.761 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:05:48.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406505', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:43:15.207984', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-29T23:05:48.842 INFO:teuthology.orchestra.console.smithi064:Power off 2023-09-29T23:05:48.843 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:05:48.867 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2023-09-29T23:05:48.868 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:05:48.893 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-09-29T23:05:50.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-29T23:05:50.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-29T23:05:50.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-29T23:05:50.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-29T23:05:52.894 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:05:53.003 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-09-29T23:05:57.006 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:05:57.116 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-09-29T23:06:01.118 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:06:01.228 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2023-09-29T23:06:01.328 INFO:teuthology.orchestra.console.smithi064:Power off completed 2023-09-29T23:06:01.433 INFO:teuthology.run:Summary data: description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} duration: 825.2760972976685 failure_reason: '"1696028408.71283 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5ff0210e998e4b3c8cb07c8e2967f887 status: fail success: false 2023-09-29T23:06:01.433 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T23:06:01.523 INFO:teuthology.run:FAIL