2024-02-03T14:26:05.115 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-03T14:26:05.116 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661 --verbose --description rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} --owner scheduled_pdiazbou@teuthology --name pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi -- /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/orig.config.yaml 2024-02-03T14:26:05.148 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-03T14:26:05.226 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661 branch: wip-pereman2-buffer-space description: rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} email: pdiazbou@redhat.com first_in_suite: false job_id: '7544661' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pereman2-buffer-space ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pereman2-buffer-space sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 owner: scheduled_pdiazbou@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1362 sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 sleep_before_teardown: 0 subset: 512/20000 suite: rados suite_branch: wip-pereman2-buffer-space suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPpZzp9pCH8Oyjw0rR2MIR8xJuJlvu1OfcdxY/MrRHXXCpOCkj6W+EKTcUzxtjaXzj9OXQA1QBr9gnoMODTd8I= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-02_19:09:44 tube: smithi user: pdiazbou verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-03T14:26:05.227 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa; will attempt to use it 2024-02-03T14:26:05.227 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks 2024-02-03T14:26:05.227 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-03T14:26:05.229 INFO:teuthology.task.internal:Checking packages... 2024-02-03T14:26:05.253 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7' 2024-02-03T14:26:05.253 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-03T14:26:05.253 INFO:teuthology.packaging:ref: None 2024-02-03T14:26:05.253 INFO:teuthology.packaging:tag: None 2024-02-03T14:26:05.253 INFO:teuthology.packaging:branch: wip-pereman2-buffer-space 2024-02-03T14:26:05.253 INFO:teuthology.packaging:sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-03T14:26:05.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pereman2-buffer-space 2024-02-03T14:26:05.534 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-432-g3cfd0d29-1jammy 2024-02-03T14:26:05.535 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-03T14:26:05.547 INFO:teuthology.task.internal:no buildpackages task found 2024-02-03T14:26:05.547 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-03T14:26:05.595 INFO:teuthology.task.internal:Saving configuration 2024-02-03T14:26:05.608 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-03T14:26:05.633 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-03T14:26:05.656 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-03 14:20:27.685066', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOJyi18tUnsRKzjkt6lmPk4qtSrg3IvNHmRBAMeEP3D0'} 2024-02-03T14:26:05.656 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-03T14:26:05.670 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-03T14:26:05.670 INFO:teuthology.run_tasks:Running task console_log... 2024-02-03T14:26:05.742 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa6ab11b160>, signals=[15]) 2024-02-03T14:26:05.743 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-03T14:26:05.754 INFO:teuthology.task.internal:Opening connections... 2024-02-03T14:26:05.754 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:26:05.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-03T14:26:05.844 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-03T14:26:05.850 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-02-03T14:26:05.956 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-02-03T14:26:05.957 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="22.04" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=jammy 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-03T14:26:06.003 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=jammy 2024-02-03T14:26:06.004 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-02-03T14:26:06.029 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-03T14:26:06.122 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-03T14:26:06.158 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-03T14:26:06.159 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-02-03T14:26:06.164 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-03T14:26:06.192 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-03T14:26:06.192 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-02-03T14:26:06.212 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-03T14:26:06.257 INFO:teuthology.run_tasks:Running task kernel... 2024-02-03T14:26:06.277 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-03T14:26:06.277 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-03T14:26:06.277 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-03T14:26:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:26:06.282 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-02-03T14:26:06.331 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-56-generic 2024-02-03T14:26:06.331 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.15.0-56-generic 2024-02-03T14:26:06.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-02-03T14:26:06.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-02-03T14:26:06.638 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-03T14:26:06.746 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-03T14:26:06.799 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-03T14:26:07.024 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,109 kB] 2024-02-03T14:26:07.163 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-03T14:26:07.239 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [393 kB] 2024-02-03T14:26:07.251 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [207 kB] 2024-02-03T14:26:07.258 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-03T14:26:07.258 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,313 kB] 2024-02-03T14:26:07.301 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,325 kB] 2024-02-03T14:26:07.316 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-03T14:26:07.317 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [214 kB] 2024-02-03T14:26:07.321 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-03T14:26:07.321 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [588 kB] 2024-02-03T14:26:07.346 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [837 kB] 2024-02-03T14:26:07.409 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB] 2024-02-03T14:26:07.411 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-03T14:26:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-03T14:26:07.431 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-03T14:26:07.432 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-03T14:26:07.432 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-03T14:26:07.593 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [560 kB] 2024-02-03T14:26:07.631 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [268 kB] 2024-02-03T14:26:07.651 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-03T14:26:07.651 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-03T14:26:07.653 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,342 kB] 2024-02-03T14:26:07.708 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [220 kB] 2024-02-03T14:26:07.714 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-03T14:26:07.879 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [686 kB] 2024-02-03T14:26:08.462 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,042 kB] 2024-02-03T14:26:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB] 2024-02-03T14:26:08.593 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-03T14:26:08.595 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-03T14:26:08.597 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,164 B] 2024-02-03T14:26:08.598 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-03T14:26:08.599 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-03T14:26:08.600 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-03T14:26:08.601 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-03T14:26:08.605 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-03T14:26:08.770 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-03T14:26:08.852 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.2 kB] 2024-02-03T14:26:08.948 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-03T14:26:08.977 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-03T14:26:09.028 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-03T14:26:13.539 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 11.2 MB in 2s (4,491 kB/s) 2024-02-03T14:26:14.846 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:26:14.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-03T14:26:14.928 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:26:15.106 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-02-03T14:26:15.106 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-02-03T14:26:15.260 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-02-03T14:26:15.260 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-03T14:26:15.260 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-03T14:26:15.261 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-03T14:26:15.261 INFO:teuthology.orchestra.run.smithi007.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-03T14:26:15.261 INFO:teuthology.orchestra.run.smithi007.stdout: python2.7-minimal 2024-02-03T14:26:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-03T14:26:15.262 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-02-03T14:26:15.262 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-03T14:26:15.262 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-generic linux-image-5.15.0-92-generic 2024-02-03T14:26:15.262 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.15.0-92-generic linux-modules-extra-5.15.0-92-generic 2024-02-03T14:26:15.263 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-02-03T14:26:15.263 INFO:teuthology.orchestra.run.smithi007.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-03T14:26:15.264 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-02-03T14:26:15.264 INFO:teuthology.orchestra.run.smithi007.stdout: thermald 2024-02-03T14:26:15.307 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-02-03T14:26:15.307 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-03T14:26:15.307 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.15.0-92-generic linux-modules-5.15.0-92-generic 2024-02-03T14:26:15.307 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.15.0-92-generic 2024-02-03T14:26:15.308 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2024-02-03T14:26:15.308 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-03T14:26:15.364 INFO:teuthology.orchestra.run.smithi007.stdout:3 upgraded, 5 newly installed, 0 to remove and 282 not upgraded. 2024-02-03T14:26:15.364 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 113 MB of archives. 2024-02-03T14:26:15.364 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-03T14:26:15.364 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-92-generic amd64 5.15.0-92.102 [22.5 MB] 2024-02-03T14:26:15.922 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-92-generic amd64 5.15.0-92.102 [11.5 MB] 2024-02-03T14:26:16.158 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-92-generic amd64 5.15.0-92.102 [63.8 MB] 2024-02-03T14:26:17.437 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.92.89 [1,700 B] 2024-02-03T14:26:17.438 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.92.89 [2,498 B] 2024-02-03T14:26:17.438 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92 all 5.15.0-92.102 [12.3 MB] 2024-02-03T14:26:17.576 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92-generic amd64 5.15.0-92.102 [2,859 kB] 2024-02-03T14:26:17.609 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.92.89 [2,346 B] 2024-02-03T14:26:18.052 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 113 MB in 2s (49.3 MB/s) 2024-02-03T14:26:18.239 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.15.0-92-generic. 2024-02-03T14:26:20.233 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-03T14:26:20.239 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-linux-modules-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-03T14:26:20.520 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:22.839 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.15.0-92-generic. 2024-02-03T14:26:22.861 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-linux-image-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-03T14:26:22.955 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:23.418 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-92-generic. 2024-02-03T14:26:23.439 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-03T14:26:23.467 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:30.115 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-linux-generic_5.15.0.92.89_amd64.deb ... 2024-02-03T14:26:30.239 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-03T14:26:30.564 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.92.89_amd64.deb ... 2024-02-03T14:26:30.701 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-03T14:26:30.987 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-92. 2024-02-03T14:26:31.010 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-linux-headers-5.15.0-92_5.15.0-92.102_all.deb ... 2024-02-03T14:26:31.054 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-03T14:26:37.046 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.15.0-92-generic. 2024-02-03T14:26:37.083 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-linux-headers-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-03T14:26:37.120 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:40.064 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.92.89_amd64.deb ... 2024-02-03T14:26:40.189 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-03T14:26:40.548 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-03T14:26:40.669 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:40.770 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-generic (5.15.0.92.89) ... 2024-02-03T14:26:40.890 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:42.340 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-03T14:26:42.341 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-03T14:26:42.341 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-92-generic 2024-02-03T14:26:42.341 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-92-generic 2024-02-03T14:26:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:44.118 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:45.544 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic (5.15.0.92.89) ... 2024-02-03T14:26:45.679 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-generic (5.15.0.92.89) ... 2024-02-03T14:26:45.822 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-03T14:26:45.884 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-03T14:26:45.884 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-92-generic 2024-02-03T14:27:00.565 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-03T14:27:00.565 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2024-02-03T14:27:00.599 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-03T14:27:00.620 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2024-02-03T14:27:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-03T14:27:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-03T14:27:01.205 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-03T14:27:01.211 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-03T14:27:01.286 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-03T14:27:01.291 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-03T14:27:01.448 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-03T14:27:01.449 INFO:teuthology.orchestra.run.smithi007.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-03T14:27:01.449 INFO:teuthology.orchestra.run.smithi007.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-03T14:27:01.483 INFO:teuthology.orchestra.run.smithi007.stdout:done 2024-02-03T14:27:01.928 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-02-03T14:27:01.995 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.92.89 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-02-03T14:27:01.996 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-02-03T14:27:01.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-92-generic 2024-02-03T14:27:01.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-03T14:27:01.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-03T14:27:01.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-03T14:27:01.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-02-03T14:27:01.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2024-02-03T14:27:02.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-02-03T14:27:02.283 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-03T14:27:02.375 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-03T14:27:02.456 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-03T14:27:02.538 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-03T14:27:03.710 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:27:03.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-03T14:27:03.791 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:27:03.971 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-02-03T14:27:03.972 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-02-03T14:27:04.125 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic is already the newest version (5.15.0.92.89). 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-03T14:27:04.126 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-03T14:27:04.164 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 282 not upgraded. 2024-02-03T14:27:04.166 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 21 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2024-02-03T14:27:04.191 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.15.0.92.89 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2024-02-03T14:27:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2024-02-03T14:27:04.193 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-02-03T14:27:04.243 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.2PDJ6IIuqD 2024-02-03T14:27:04.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2PDJ6IIuqD 2024-02-03T14:27:04.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.2PDJ6IIuqD 2024-02-03T14:27:04.426 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.2PDJ6IIuqD is 10KB 2024-02-03T14:27:04.446 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.2PDJ6IIuqD 2024-02-03T14:27:04.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-03T14:27:04.507 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:27:04.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-03T14:27:04.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-03T14:27:04.588 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-92-generic 2024-02-03T14:27:04.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2024-02-03T14:27:05.860 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2024-02-03T14:27:05.861 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-03T14:27:05.863 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-02-03T14:27:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-03T14:27:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-03T14:27:06.264 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-03T14:27:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-03T14:27:06.312 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-03T14:27:06.317 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-03T14:27:06.447 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-03T14:27:06.448 INFO:teuthology.orchestra.run.smithi007.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-03T14:27:06.448 INFO:teuthology.orchestra.run.smithi007.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-03T14:27:06.469 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-02-03T14:27:06.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-02-03T14:27:36.518 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-03T14:27:36.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-03T14:27:36.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-03T14:28:36.581 DEBUG:teuthology.orchestra.remote:timed out 2024-02-03T14:28:45.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-03T14:28:45.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-03T14:28:45.945 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-02-03T14:28:46.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-03T14:28:46.685 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-92-generic"... 2024-02-03T14:28:46.685 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-02-03T14:28:46.691 INFO:teuthology.orchestra.run.smithi007.stdout:5.15.0-92-generic 2024-02-03T14:28:46.691 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-92-generic vs 5.15.0-92-generic 2024-02-03T14:28:46.691 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-03T14:28:46.691 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-03T14:28:47.693 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-03T14:28:47.693 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-03T14:28:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-02-03T14:28:47.808 DEBUG:teuthology.parallel:result is None 2024-02-03T14:28:47.809 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-03T14:28:47.816 INFO:teuthology.task.internal:Creating test directory... 2024-02-03T14:28:47.816 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-03T14:28:47.822 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-03T14:28:47.859 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-03T14:28:47.909 INFO:teuthology.task.internal:Creating archive directory... 2024-02-03T14:28:47.909 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-03T14:28:47.927 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-03T14:28:47.951 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-03T14:28:47.952 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-03T14:28:47.990 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-03T14:28:47.999 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-03T14:28:48.001 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-03T14:28:48.009 INFO:teuthology.task.internal:Configuring sudo... 2024-02-03T14:28:48.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-03T14:28:48.055 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-03T14:28:48.064 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-03T14:28:48.065 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-03T14:28:48.107 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-03T14:28:48.155 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-03T14:28:48.203 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:28:48.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-03T14:28:48.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-02-03T14:28:48.460 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-03T14:28:48.467 INFO:teuthology.task.internal:Starting timer... 2024-02-03T14:28:48.468 INFO:teuthology.run_tasks:Running task pcp... 2024-02-03T14:28:48.496 INFO:teuthology.run_tasks:Running task selinux... 2024-02-03T14:28:48.538 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2024-02-03T14:28:48.538 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-03T14:28:48.539 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-03T14:28:48.539 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-03T14:28:48.539 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-03T14:28:48.587 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-03T14:28:48.612 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-03T14:28:48.613 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-03T14:35:47.378 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com')] 2024-02-03T14:35:47.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-03T14:35:47.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-03T14:35:47.467 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-02-03T14:35:47.545 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-02-03T14:35:47.545 INFO:teuthology.run_tasks:Running task clock... 2024-02-03T14:35:47.557 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-03T14:35:47.557 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-03T14:35:47.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Command line: ntpd -gq 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: ---------------------------------------------------- 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: ntp-4 is maintained by Network Time Foundation, 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: corporation. Support and training for ntp-4 are 2024-02-03T14:35:47.617 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: available at https://www.nwtime.org/support 2024-02-03T14:35:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: ---------------------------------------------------- 2024-02-03T14:35:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: proto: precision = 0.045 usec (-24) 2024-02-03T14:35:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: basedate set to 2022-02-04 2024-02-03T14:35:47.618 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: gps base set to 2022-02-06 (week 2196) 2024-02-03T14:35:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-03T14:35:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-03T14:35:47.619 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: restrict ::: KOD does nothing without LIMITED. 2024-02-03T14:35:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-03T14:35:47.620 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen and drop on 0 v6wildcard [::]:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen normally on 2 lo 127.0.0.1:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen normally on 3 ens1f0 172.21.15.7:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen normally on 4 lo [::1]:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2024-02-03T14:35:47.621 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:47 ntpd[16986]: Listening on routing socket on fd #22 for interface updates 2024-02-03T14:35:49.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:49.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:49.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:49.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:49.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:49.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:49 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:50.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:50 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:51.620 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:51.620 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:51.621 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:51 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:52.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:52.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:52.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:52.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:52.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:52.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:52 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:53.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:53.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:53 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:54.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:54 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: can't open /var/log/ntpstats/rawstats.20240203: Permission denied 2024-02-03T14:35:55.618 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-03T14:35:55.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: can't open /var/log/ntpstats/peerstats.20240203: Permission denied 2024-02-03T14:35:55.619 INFO:teuthology.orchestra.run.smithi007.stdout: 3 Feb 14:35:55 ntpd[16986]: ntpd: time slew -0.000360 s 2024-02-03T14:35:55.619 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew -0.000360s 2024-02-03T14:35:55.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-03T14:35:55.619 INFO:teuthology.orchestra.run.smithi007.stderr: 3 Feb 14:35:55 ntpd[16986]: can't open /var/log/ntpstats/loopstats.20240203: Permission denied 2024-02-03T14:35:55.670 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-02-03T14:35:55.671 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-02-03T14:35:55.671 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-03T14:35:55.671 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-03T14:35:55.671 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-03T14:35:55.671 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-03T14:35:55.672 INFO:teuthology.run_tasks:Running task install... 2024-02-03T14:35:55.680 DEBUG:teuthology.task.install:project ceph 2024-02-03T14:35:55.681 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'}} 2024-02-03T14:35:55.681 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'} 2024-02-03T14:35:55.681 INFO:teuthology.task.install:Using flavor: default 2024-02-03T14:35:55.686 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-03T14:35:55.687 INFO:teuthology.task.install:extra packages: [] 2024-02-03T14:35:55.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-key list | grep Ceph 2024-02-03T14:35:55.773 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-03T14:35:55.800 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-03T14:35:55.801 INFO:teuthology.orchestra.run.smithi007.stdout:uid [ unknown] Ceph.com (release key) 2024-02-03T14:35:55.802 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-03T14:35:55.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-03T14:35:55.955 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default/ 2024-02-03T14:35:55.955 INFO:teuthology.task.install.deb:Package version is 19.0.0-432-g3cfd0d29-1jammy 2024-02-03T14:35:56.074 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:35:56.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-03T14:35:56.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2024-02-03T14:35:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-03T14:35:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-03T14:35:56.291 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-03T14:35:56.312 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-03T14:35:56.330 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy InRelease 2024-02-03T14:35:56.351 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-02-03T14:35:56.372 INFO:teuthology.orchestra.run.smithi007.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-03T14:35:56.394 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2024-02-03T14:35:56.421 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-03T14:35:56.655 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 28.8 kB in 0s (77.5 kB/s) 2024-02-03T14:35:57.555 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:35:57.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-432-g3cfd0d29-1jammy cephadm=19.0.0-432-g3cfd0d29-1jammy ceph-mds=19.0.0-432-g3cfd0d29-1jammy ceph-mgr=19.0.0-432-g3cfd0d29-1jammy ceph-common=19.0.0-432-g3cfd0d29-1jammy ceph-fuse=19.0.0-432-g3cfd0d29-1jammy ceph-test=19.0.0-432-g3cfd0d29-1jammy ceph-volume=19.0.0-432-g3cfd0d29-1jammy radosgw=19.0.0-432-g3cfd0d29-1jammy python3-rados=19.0.0-432-g3cfd0d29-1jammy python3-rgw=19.0.0-432-g3cfd0d29-1jammy python3-cephfs=19.0.0-432-g3cfd0d29-1jammy python3-rbd=19.0.0-432-g3cfd0d29-1jammy libcephfs2=19.0.0-432-g3cfd0d29-1jammy libcephfs-dev=19.0.0-432-g3cfd0d29-1jammy librados2=19.0.0-432-g3cfd0d29-1jammy librbd1=19.0.0-432-g3cfd0d29-1jammy rbd-fuse=19.0.0-432-g3cfd0d29-1jammy 2024-02-03T14:35:57.624 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2024-02-03T14:35:57.796 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2024-02-03T14:35:57.796 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2024-02-03T14:35:57.960 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages were automatically installed and are no longer required: 2024-02-03T14:35:57.960 INFO:teuthology.orchestra.run.smithi007.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-03T14:35:57.960 INFO:teuthology.orchestra.run.smithi007.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-03T14:35:57.960 INFO:teuthology.orchestra.run.smithi007.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-03T14:35:57.961 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-03T14:35:57.961 INFO:teuthology.orchestra.run.smithi007.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-03T14:35:57.961 INFO:teuthology.orchestra.run.smithi007.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-03T14:35:57.961 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2024-02-03T14:35:57.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-03T14:35:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-03T14:35:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-03T14:35:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-03T14:35:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-03T14:35:57.962 INFO:teuthology.orchestra.run.smithi007.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-03T14:35:57.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-03T14:35:57.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet zip 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-03T14:35:57.965 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2024-02-03T14:35:57.966 INFO:teuthology.orchestra.run.smithi007.stdout: btrfs-tools 2024-02-03T14:35:58.063 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2024-02-03T14:35:58.063 INFO:teuthology.orchestra.run.smithi007.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-03T14:35:58.063 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-03T14:35:58.063 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-03T14:35:58.063 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-03T14:35:58.064 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-03T14:35:58.065 INFO:teuthology.orchestra.run.smithi007.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-03T14:35:58.129 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 118 newly installed, 0 to remove and 269 not upgraded. 2024-02-03T14:35:58.130 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 172 MB of archives. 2024-02-03T14:35:58.130 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 712 MB of additional disk space will be used. 2024-02-03T14:35:58.130 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-03T14:35:58.237 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-03T14:35:58.240 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-03T14:35:58.260 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-03T14:35:58.284 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-432-g3cfd0d29-1jammy [3,514 kB] 2024-02-03T14:35:58.333 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-03T14:35:58.338 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-03T14:35:58.352 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-03T14:35:58.356 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-03T14:35:58.357 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-03T14:35:58.358 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-03T14:35:58.367 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-03T14:35:58.369 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-03T14:35:58.373 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-03T14:35:58.378 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-03T14:35:58.398 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-03T14:35:58.401 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-02-03T14:35:58.403 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-03T14:35:58.405 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-03T14:35:58.406 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-03T14:35:58.406 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-03T14:35:58.408 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-03T14:35:58.408 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-03T14:35:58.408 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-03T14:35:58.427 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-03T14:35:58.427 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-03T14:35:58.447 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-03T14:35:58.447 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-03T14:35:58.450 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-03T14:35:58.451 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-03T14:35:58.451 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-03T14:35:58.452 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-03T14:35:58.453 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-03T14:35:58.453 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-03T14:35:58.468 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-03T14:35:58.469 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-03T14:35:58.488 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-03T14:35:58.489 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-03T14:35:58.494 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-03T14:35:58.495 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-03T14:35:58.495 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-03T14:35:58.496 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-03T14:35:58.497 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-03T14:35:58.499 INFO:teuthology.orchestra.run.smithi007.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-03T14:35:58.508 INFO:teuthology.orchestra.run.smithi007.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-03T14:35:58.510 INFO:teuthology.orchestra.run.smithi007.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-03T14:35:58.521 INFO:teuthology.orchestra.run.smithi007.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-03T14:35:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-03T14:35:58.524 INFO:teuthology.orchestra.run.smithi007.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-03T14:35:58.525 INFO:teuthology.orchestra.run.smithi007.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-03T14:35:58.527 INFO:teuthology.orchestra.run.smithi007.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-03T14:35:58.585 INFO:teuthology.orchestra.run.smithi007.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-03T14:35:58.586 INFO:teuthology.orchestra.run.smithi007.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-03T14:35:58.587 INFO:teuthology.orchestra.run.smithi007.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-03T14:35:58.587 INFO:teuthology.orchestra.run.smithi007.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-03T14:35:58.588 INFO:teuthology.orchestra.run.smithi007.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-03T14:35:58.606 INFO:teuthology.orchestra.run.smithi007.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-03T14:35:58.608 INFO:teuthology.orchestra.run.smithi007.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-03T14:35:58.608 INFO:teuthology.orchestra.run.smithi007.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-03T14:35:58.627 INFO:teuthology.orchestra.run.smithi007.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-03T14:35:58.627 INFO:teuthology.orchestra.run.smithi007.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-03T14:35:58.635 INFO:teuthology.orchestra.run.smithi007.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-03T14:35:58.636 INFO:teuthology.orchestra.run.smithi007.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-03T14:35:58.636 INFO:teuthology.orchestra.run.smithi007.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-03T14:35:58.637 INFO:teuthology.orchestra.run.smithi007.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-03T14:35:58.637 INFO:teuthology.orchestra.run.smithi007.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-03T14:35:58.638 INFO:teuthology.orchestra.run.smithi007.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-03T14:35:58.641 INFO:teuthology.orchestra.run.smithi007.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-03T14:35:58.656 INFO:teuthology.orchestra.run.smithi007.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-03T14:35:58.657 INFO:teuthology.orchestra.run.smithi007.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-03T14:35:58.658 INFO:teuthology.orchestra.run.smithi007.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-03T14:35:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-03T14:35:58.679 INFO:teuthology.orchestra.run.smithi007.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-03T14:35:58.679 INFO:teuthology.orchestra.run.smithi007.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-03T14:35:58.680 INFO:teuthology.orchestra.run.smithi007.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-03T14:35:58.685 INFO:teuthology.orchestra.run.smithi007.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-03T14:35:58.686 INFO:teuthology.orchestra.run.smithi007.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-03T14:35:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-03T14:35:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-03T14:35:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-03T14:35:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-03T14:35:58.717 INFO:teuthology.orchestra.run.smithi007.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-03T14:35:58.718 INFO:teuthology.orchestra.run.smithi007.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-03T14:35:58.727 INFO:teuthology.orchestra.run.smithi007.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-03T14:35:58.727 INFO:teuthology.orchestra.run.smithi007.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-03T14:35:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-03T14:35:58.730 INFO:teuthology.orchestra.run.smithi007.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-03T14:35:58.731 INFO:teuthology.orchestra.run.smithi007.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-03T14:35:59.241 INFO:teuthology.orchestra.run.smithi007.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-432-g3cfd0d29-1jammy [3,207 kB] 2024-02-03T14:35:59.573 INFO:teuthology.orchestra.run.smithi007.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-432-g3cfd0d29-1jammy [710 kB] 2024-02-03T14:35:59.641 INFO:teuthology.orchestra.run.smithi007.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-432-g3cfd0d29-1jammy [356 kB] 2024-02-03T14:35:59.675 INFO:teuthology.orchestra.run.smithi007.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-432-g3cfd0d29-1jammy [32.8 kB] 2024-02-03T14:35:59.677 INFO:teuthology.orchestra.run.smithi007.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-432-g3cfd0d29-1jammy [180 kB] 2024-02-03T14:35:59.681 INFO:teuthology.orchestra.run.smithi007.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-432-g3cfd0d29-1jammy [64.1 kB] 2024-02-03T14:35:59.683 INFO:teuthology.orchestra.run.smithi007.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-432-g3cfd0d29-1jammy [330 kB] 2024-02-03T14:35:59.715 INFO:teuthology.orchestra.run.smithi007.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-432-g3cfd0d29-1jammy [6,219 kB] 2024-02-03T14:36:00.075 INFO:teuthology.orchestra.run.smithi007.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-432-g3cfd0d29-1jammy [112 kB] 2024-02-03T14:36:00.077 INFO:teuthology.orchestra.run.smithi007.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-432-g3cfd0d29-1jammy [465 kB] 2024-02-03T14:36:00.104 INFO:teuthology.orchestra.run.smithi007.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-432-g3cfd0d29-1jammy [23.0 MB] 2024-02-03T14:36:00.953 INFO:teuthology.orchestra.run.smithi007.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-432-g3cfd0d29-1jammy [5,056 kB] 2024-02-03T14:36:01.047 INFO:teuthology.orchestra.run.smithi007.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-432-g3cfd0d29-1jammy [238 kB] 2024-02-03T14:36:01.053 INFO:teuthology.orchestra.run.smithi007.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-432-g3cfd0d29-1jammy [124 kB] 2024-02-03T14:36:01.057 INFO:teuthology.orchestra.run.smithi007.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-432-g3cfd0d29-1jammy [1,518 kB] 2024-02-03T14:36:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-432-g3cfd0d29-1jammy [6,237 kB] 2024-02-03T14:36:01.313 INFO:teuthology.orchestra.run.smithi007.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-432-g3cfd0d29-1jammy [22.7 MB] 2024-02-03T14:36:02.184 INFO:teuthology.orchestra.run.smithi007.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-432-g3cfd0d29-1jammy [14.0 kB] 2024-02-03T14:36:02.186 INFO:teuthology.orchestra.run.smithi007.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-432-g3cfd0d29-1jammy [865 kB] 2024-02-03T14:36:02.203 INFO:teuthology.orchestra.run.smithi007.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-432-g3cfd0d29-1jammy [2,181 kB] 2024-02-03T14:36:02.242 INFO:teuthology.orchestra.run.smithi007.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-432-g3cfd0d29-1jammy [837 kB] 2024-02-03T14:36:02.254 INFO:teuthology.orchestra.run.smithi007.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-432-g3cfd0d29-1jammy [139 kB] 2024-02-03T14:36:02.256 INFO:teuthology.orchestra.run.smithi007.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-432-g3cfd0d29-1jammy [1,906 kB] 2024-02-03T14:36:02.282 INFO:teuthology.orchestra.run.smithi007.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-432-g3cfd0d29-1jammy [8,625 kB] 2024-02-03T14:36:02.568 INFO:teuthology.orchestra.run.smithi007.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-432-g3cfd0d29-1jammy [14.2 kB] 2024-02-03T14:36:02.568 INFO:teuthology.orchestra.run.smithi007.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-432-g3cfd0d29-1jammy [48.3 MB] 2024-02-03T14:36:04.339 INFO:teuthology.orchestra.run.smithi007.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-432-g3cfd0d29-1jammy [121 kB] 2024-02-03T14:36:04.419 INFO:teuthology.orchestra.run.smithi007.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-432-g3cfd0d29-1jammy [38.4 kB] 2024-02-03T14:36:04.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-432-g3cfd0d29-1jammy [12.3 MB] 2024-02-03T14:36:04.739 INFO:teuthology.orchestra.run.smithi007.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-432-g3cfd0d29-1jammy [91.7 kB] 2024-02-03T14:36:05.323 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 172 MB in 7s (25.8 MB/s) 2024-02-03T14:36:05.496 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-03T14:36:05.598 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2024-02-03T14:36:05.604 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-03T14:36:05.694 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-03T14:36:06.006 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-03T14:36:06.033 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-03T14:36:06.065 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-03T14:36:06.501 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-03T14:36:06.527 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-03T14:36:06.568 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-03T14:36:07.030 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-03T14:36:07.056 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-03T14:36:07.105 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:36:07.441 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-03T14:36:07.468 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-03T14:36:07.499 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:36:07.836 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-03T14:36:07.862 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-03T14:36:07.904 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:36:08.256 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-03T14:36:08.282 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-03T14:36:08.322 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-03T14:36:08.625 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librados2. 2024-02-03T14:36:08.652 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../007-librados2_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:08.683 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librados2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:09.153 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librbd1. 2024-02-03T14:36:09.180 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../008-librbd1_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:09.219 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librbd1 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:09.622 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs2. 2024-02-03T14:36:09.640 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:09.689 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:09.950 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rados. 2024-02-03T14:36:09.977 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../010-python3-rados_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:10.017 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rados (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:10.319 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-03T14:36:10.347 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:10.378 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-argparse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:10.614 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cephfs. 2024-02-03T14:36:10.630 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:10.672 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cephfs (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:11.192 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-03T14:36:11.219 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:11.258 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-ceph-common (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:11.570 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-03T14:36:11.597 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-03T14:36:11.637 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-03T14:36:11.915 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-prettytable. 2024-02-03T14:36:11.941 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-03T14:36:11.973 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-03T14:36:12.217 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rbd. 2024-02-03T14:36:12.233 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:12.276 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rbd (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:12.595 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-03T14:36:12.622 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-03T14:36:12.670 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-03T14:36:12.940 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-03T14:36:12.966 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-03T14:36:13.006 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-03T14:36:13.284 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-03T14:36:13.300 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-03T14:36:13.334 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-03T14:36:13.637 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua5.1. 2024-02-03T14:36:13.664 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-03T14:36:13.704 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-03T14:36:14.015 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-any. 2024-02-03T14:36:14.031 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-03T14:36:14.074 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-03T14:36:14.293 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package zip. 2024-02-03T14:36:14.309 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-03T14:36:14.343 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking zip (3.0-12build2) ... 2024-02-03T14:36:14.630 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package unzip. 2024-02-03T14:36:14.656 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-02-03T14:36:14.696 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-02-03T14:36:15.109 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package luarocks. 2024-02-03T14:36:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-03T14:36:15.175 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-03T14:36:15.550 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package librgw2. 2024-02-03T14:36:15.577 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../025-librgw2_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:15.616 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking librgw2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:16.278 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rgw. 2024-02-03T14:36:16.304 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:16.498 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rgw (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:16.956 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-03T14:36:16.984 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-03T14:36:17.014 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-03T14:36:17.325 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libradosstriper1. 2024-02-03T14:36:17.352 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:17.393 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libradosstriper1 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:17.820 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-common. 2024-02-03T14:36:17.847 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../029-ceph-common_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:17.887 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-common (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:19.259 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-base. 2024-02-03T14:36:19.286 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../030-ceph-base_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:19.326 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-base (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:19.930 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-03T14:36:19.957 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-03T14:36:19.997 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-03T14:36:20.274 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cheroot. 2024-02-03T14:36:20.301 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-03T14:36:20.333 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-03T14:36:20.619 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-03T14:36:20.646 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-03T14:36:20.677 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-03T14:36:20.980 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-03T14:36:21.007 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-03T14:36:21.047 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-03T14:36:21.350 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-03T14:36:21.377 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-03T14:36:21.417 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-03T14:36:21.678 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempora. 2024-02-03T14:36:21.694 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-03T14:36:21.728 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-03T14:36:22.031 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-portend. 2024-02-03T14:36:22.058 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-03T14:36:22.098 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-03T14:36:22.401 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-03T14:36:22.428 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-03T14:36:22.467 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-03T14:36:22.745 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-03T14:36:22.772 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-03T14:36:22.812 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-03T14:36:23.149 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-natsort. 2024-02-03T14:36:23.176 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-03T14:36:23.215 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-03T14:36:23.560 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-logutils. 2024-02-03T14:36:23.587 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-03T14:36:23.627 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-03T14:36:23.930 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-03T14:36:23.957 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-03T14:36:23.996 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-03T14:36:24.274 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-mako. 2024-02-03T14:36:24.302 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-03T14:36:24.333 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-03T14:36:24.619 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-03T14:36:24.646 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-03T14:36:24.678 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-03T14:36:24.947 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-03T14:36:24.974 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-03T14:36:25.022 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-03T14:36:25.317 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webob. 2024-02-03T14:36:25.343 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-03T14:36:25.383 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-03T14:36:25.711 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-waitress. 2024-02-03T14:36:25.738 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-03T14:36:25.781 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-03T14:36:26.098 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-tempita. 2024-02-03T14:36:26.124 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-03T14:36:26.164 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-03T14:36:26.467 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-paste. 2024-02-03T14:36:26.494 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-03T14:36:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-03T14:36:26.846 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-03T14:36:26.861 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-03T14:36:27.095 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-03T14:36:27.415 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-03T14:36:27.442 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-03T14:36:27.481 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-03T14:36:27.751 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-webtest. 2024-02-03T14:36:27.767 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-03T14:36:27.801 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-03T14:36:28.104 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pecan. 2024-02-03T14:36:28.131 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-03T14:36:28.171 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-03T14:36:28.509 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-certifi. 2024-02-03T14:36:28.536 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-03T14:36:28.592 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-03T14:36:28.937 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-idna. 2024-02-03T14:36:28.964 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-03T14:36:29.003 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-03T14:36:29.267 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-urllib3. 2024-02-03T14:36:29.294 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-03T14:36:29.333 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-03T14:36:29.603 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests. 2024-02-03T14:36:29.630 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-03T14:36:29.672 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-03T14:36:29.972 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-03T14:36:29.999 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-03T14:36:30.039 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-03T14:36:30.300 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-03T14:36:30.316 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:30.350 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-modules-core (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:30.730 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-03T14:36:30.757 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:30.796 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:31.099 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-03T14:36:31.126 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-03T14:36:31.166 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-03T14:36:31.444 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-openssl. 2024-02-03T14:36:31.460 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-03T14:36:31.652 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-03T14:36:32.706 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr. 2024-02-03T14:36:32.733 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:32.772 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:33.183 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mon. 2024-02-03T14:36:33.211 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:33.250 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mon (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:33.962 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-osd. 2024-02-03T14:36:33.989 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:34.028 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-osd (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:35.081 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph. 2024-02-03T14:36:35.108 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../066-ceph_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:35.147 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:35.467 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-fuse. 2024-02-03T14:36:35.494 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:35.534 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-fuse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:35.920 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mds. 2024-02-03T14:36:35.947 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:35.986 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mds (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:36.331 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cephadm. 2024-02-03T14:36:36.347 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../069-cephadm_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:36.380 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cephadm (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:36.684 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-03T14:36:36.711 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-03T14:36:36.750 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-03T14:36:37.044 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-03T14:36:37.061 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-03T14:36:37.102 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-03T14:36:38.116 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-babel. 2024-02-03T14:36:38.143 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-03T14:36:38.182 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-03T14:36:38.468 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-jinja2. 2024-02-03T14:36:38.484 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-03T14:36:38.518 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-03T14:36:38.838 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-03T14:36:38.865 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:38.904 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-cephadm (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:39.249 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-03T14:36:39.265 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-03T14:36:39.307 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-03T14:36:39.569 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-routes. 2024-02-03T14:36:39.596 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-03T14:36:39.635 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-03T14:36:39.938 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-03T14:36:39.954 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:39.988 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-dashboard (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:40.383 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-03T14:36:40.399 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-03T14:36:40.449 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-03T14:36:40.945 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-joblib. 2024-02-03T14:36:40.972 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-03T14:36:41.011 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-03T14:36:41.373 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-03T14:36:41.400 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-03T14:36:41.439 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-03T14:36:41.725 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-sklearn. 2024-02-03T14:36:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-03T14:36:41.792 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-03T14:36:42.525 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-03T14:36:42.553 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:43.780 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-cachetools. 2024-02-03T14:36:43.811 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-03T14:36:43.847 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-03T14:36:44.124 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-03T14:36:44.152 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-03T14:36:44.182 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-03T14:36:44.478 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-03T14:36:44.506 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-03T14:36:44.544 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-03T14:36:44.847 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-rsa. 2024-02-03T14:36:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-03T14:36:44.913 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-03T14:36:45.234 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-google-auth. 2024-02-03T14:36:45.262 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-03T14:36:45.300 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-03T14:36:45.621 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-03T14:36:45.649 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-03T14:36:45.687 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-03T14:36:45.965 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-websocket. 2024-02-03T14:36:45.992 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-03T14:36:46.023 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-03T14:36:46.284 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-03T14:36:46.310 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-03T14:36:46.351 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-03T14:36:46.981 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-03T14:36:46.997 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:47.031 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:47.351 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-03T14:36:47.379 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-03T14:36:47.417 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-03T14:36:47.770 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-03T14:36:47.798 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-03T14:36:47.837 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-03T14:36:48.081 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package jq. 2024-02-03T14:36:48.097 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-03T14:36:48.156 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-03T14:36:48.358 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2024-02-03T14:36:48.375 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-03T14:36:48.417 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-03T14:36:48.745 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package xmlstarlet. 2024-02-03T14:36:48.761 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-03T14:36:48.803 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-03T14:36:49.116 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-test. 2024-02-03T14:36:49.133 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../097-ceph-test_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:49.173 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-test (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:51.353 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ceph-volume. 2024-02-03T14:36:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-432-g3cfd0d29-1jammy_all.deb ... 2024-02-03T14:36:51.419 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ceph-volume (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:51.723 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-03T14:36:51.751 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:51.789 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libcephfs-dev (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:36:52.100 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-03T14:36:52.128 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-03T14:36:52.167 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-03T14:36:52.688 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-03T14:36:52.718 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-03T14:36:52.754 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-03T14:36:53.125 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package nvme-cli. 2024-02-03T14:36:53.153 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-03T14:36:53.200 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-03T14:36:53.605 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package pkg-config. 2024-02-03T14:36:53.633 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-03T14:36:53.671 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-03T14:36:54.141 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-03T14:36:54.157 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-03T14:36:54.199 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-03T14:36:54.586 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-03T14:36:54.614 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-03T14:36:54.652 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-03T14:36:54.955 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pastescript. 2024-02-03T14:36:54.982 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-03T14:36:55.021 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-03T14:36:55.308 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pluggy. 2024-02-03T14:36:55.335 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-03T14:36:55.366 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-03T14:36:55.661 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-psutil. 2024-02-03T14:36:55.689 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-03T14:36:55.727 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-03T14:36:56.080 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-py. 2024-02-03T14:36:56.108 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-03T14:36:56.155 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-03T14:36:56.500 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pygments. 2024-02-03T14:36:56.528 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-03T14:36:56.566 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-03T14:36:57.020 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-03T14:36:57.049 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-03T14:36:57.087 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-03T14:36:57.364 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-toml. 2024-02-03T14:36:57.393 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-03T14:36:57.431 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-03T14:36:57.733 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-pytest. 2024-02-03T14:36:57.763 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-03T14:36:57.800 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-03T14:36:58.112 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package python3-simplejson. 2024-02-03T14:36:58.128 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-03T14:36:58.162 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-03T14:36:58.431 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-03T14:36:58.448 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-03T14:36:58.489 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-03T14:36:59.218 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package radosgw. 2024-02-03T14:36:59.247 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../116-radosgw_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:36:59.284 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking radosgw (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:00.072 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package rbd-fuse. 2024-02-03T14:37:00.087 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-432-g3cfd0d29-1jammy_amd64.deb ... 2024-02-03T14:37:00.121 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking rbd-fuse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:00.417 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-03T14:37:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-03T14:37:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-03T14:37:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-03T14:37:01.464 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-03T14:37:01.903 INFO:teuthology.orchestra.run.smithi007.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-03T14:37:01.970 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cephadm (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:02.488 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user cephadm....done 2024-02-03T14:37:02.634 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-03T14:37:02.902 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-03T14:37:03.153 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-03T14:37:03.279 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-03T14:37:03.539 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-03T14:37:03.807 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-03T14:37:03.907 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-03T14:37:04.226 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-03T14:37:04.612 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-03T14:37:04.871 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-03T14:37:05.156 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-02-03T14:37:05.399 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-03T14:37:05.641 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-03T14:37:05.901 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-03T14:37:06.152 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-argparse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:06.436 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-03T14:37:06.562 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-03T14:37:06.688 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-03T14:37:06.814 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-03T14:37:07.074 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-03T14:37:07.461 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-03T14:37:07.497 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-03T14:37:07.552 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-03T14:37:07.661 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-03T14:37:08.029 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-03T14:37:08.155 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-03T14:37:08.431 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-03T14:37:08.707 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-03T14:37:09.008 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up zip (3.0-12build2) ... 2024-02-03T14:37:09.109 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-03T14:37:09.828 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-03T14:37:10.095 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-03T14:37:10.221 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-03T14:37:10.347 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-03T14:37:10.657 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-03T14:37:11.041 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-03T14:37:11.301 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-03T14:37:11.669 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-03T14:37:11.953 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-03T14:37:12.305 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-03T14:37:12.564 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-03T14:37:12.690 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-ceph-common (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:12.999 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-idna (3.3-1) ... 2024-02-03T14:37:13.328 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-03T14:37:14.843 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-03T14:37:15.219 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:37:15.353 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-03T14:37:15.638 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-03T14:37:15.764 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-03T14:37:15.890 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-03T14:37:16.191 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-03T14:37:16.434 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-03T14:37:16.694 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:37:16.853 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-03T14:37:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-03T14:37:17.438 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-03T14:37:17.706 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-03T14:37:17.966 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-03T14:37:18.200 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-03T14:37:18.451 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-03T14:37:18.586 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-03T14:37:18.879 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-03T14:37:19.005 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-03T14:37:19.306 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-03T14:37:19.607 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-03T14:37:19.850 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-03T14:37:19.951 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-03T14:37:20.069 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-03T14:37:20.166 INFO:teuthology.orchestra.run.smithi007.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-03T14:37:20.404 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-03T14:37:20.530 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-03T14:37:20.923 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-03T14:37:21.191 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-03T14:37:21.300 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-03T14:37:21.543 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-03T14:37:21.669 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-03T14:37:21.987 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-03T14:37:22.297 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-03T14:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-03T14:37:22.707 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-03T14:37:22.975 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-03T14:37:23.242 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-03T14:37:23.571 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-03T14:37:23.965 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-03T14:37:24.233 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-03T14:37:24.560 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-03T14:37:24.687 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librados2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:24.813 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libsqlite3-mod-ceph (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:24.939 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-03T14:37:25.308 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:25.418 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libradosstriper1 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:25.536 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-03T14:37:25.812 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librbd1 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:25.938 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-modules-core (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:26.056 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-fuse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:26.249 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-03T14:37:26.249 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-03T14:37:26.757 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libcephfs-dev (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:26.875 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rados (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:26.984 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up librgw2 (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:27.111 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-03T14:37:28.323 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rbd (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:28.449 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up rbd-fuse (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:28.566 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-rgw (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:28.667 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up python3-cephfs (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:28.802 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-common (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:29.370 INFO:teuthology.orchestra.run.smithi007.stdout:Adding group ceph....done 2024-02-03T14:37:29.696 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user ceph....done 2024-02-03T14:37:29.874 INFO:teuthology.orchestra.run.smithi007.stdout:Setting system user ceph properties....done 2024-02-03T14:37:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-03T14:37:30.032 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-03T14:37:30.408 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-03T14:37:30.862 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-test (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:30.988 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up radosgw (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:31.462 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-03T14:37:31.463 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-03T14:37:31.948 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-base (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:32.316 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-03T14:37:32.817 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mds (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:33.048 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-03T14:37:33.048 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-03T14:37:33.527 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:33.762 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-03T14:37:33.762 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-03T14:37:34.270 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-osd (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:34.557 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-03T14:37:34.557 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-03T14:37:35.096 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-k8sevents (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:35.222 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:35.374 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mon (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:35.593 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-03T14:37:35.593 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-03T14:37:36.088 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-cephadm (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:36.239 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:36.365 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-mgr-dashboard (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:36.568 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ceph-volume (19.0.0-432-g3cfd0d29-1jammy) ... 2024-02-03T14:37:37.036 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-03T14:37:37.154 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-03T14:37:37.306 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-03T14:37:38.399 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-03T14:37:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-03T14:37:39.067 DEBUG:teuthology.parallel:result is None 2024-02-03T14:37:39.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-03T14:37:39.210 DEBUG:teuthology.orchestra.run.smithi007:> dpkg-query -W -f '${Version}' ceph 2024-02-03T14:37:39.229 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-432-g3cfd0d29-1jammy 2024-02-03T14:37:39.230 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-432-g3cfd0d29-1jammy 2024-02-03T14:37:39.230 INFO:teuthology.task.install:The correct ceph version 19.0.0-432-g3cfd0d29-1jammy is installed. 2024-02-03T14:37:39.231 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-03T14:37:39.232 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:39.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-03T14:37:39.283 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-03T14:37:39.284 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:39.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-02-03T14:37:39.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-03T14:37:39.399 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-03T14:37:39.399 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:39.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-03T14:37:39.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-03T14:37:39.510 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-03T14:37:39.511 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:39.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-02-03T14:37:39.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-03T14:37:39.623 INFO:teuthology.run_tasks:Running task ceph... 2024-02-03T14:37:39.720 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-03T14:37:39.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-02-03T14:37:39.733 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-03T14:37:39.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-03T14:37:39.787 INFO:tasks.ceph:Creating extra log directories... 2024-02-03T14:37:39.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-03T14:37:39.854 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-03T14:37:39.854 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-02-03T14:37:39.854 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661', 'branch': 'wip-pereman2-buffer-space', 'description': 'rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}', 'email': 'pdiazbou@redhat.com', 'first_in_suite': False, 'job_id': '7544661', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pereman2-buffer-space'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pereman2-buffer-space', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'}}, 'owner': 'scheduled_pdiazbou@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1362, 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7', 'sleep_before_teardown': 0, 'subset': '512/20000', 'suite': 'rados', 'suite_branch': 'wip-pereman2-buffer-space', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPpZzp9pCH8Oyjw0rR2MIR8xJuJlvu1OfcdxY/MrRHXXCpOCkj6W+EKTcUzxtjaXzj9OXQA1QBr9gnoMODTd8I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool', 'mon/pool_ops.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd9fdb2209e15b39d9f061fd85399f352ce0f0894', 'timestamp': '2024-02-02_19:09:44', 'tube': 'smithi', 'user': 'pdiazbou', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-03T14:37:39.855 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-03T14:37:39.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-02-03T14:37:39.949 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:39.950 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-02-03T14:37:39.996 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-03T14:37:39.996 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-02-03T14:37:40.044 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-03T14:37:40.044 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-03T14:37:40.044 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-02-03T14:37:40.044 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-03T14:37:40.045 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-03 14:32:30.456160848 +0000 2024-02-03T14:37:40.045 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-03 14:32:30.188163239 +0000 2024-02-03T14:37:40.045 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-03 14:32:30.188163239 +0000 2024-02-03T14:37:40.045 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-02-03T14:37:40.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-03T14:37:40.099 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-03T14:37:40.099 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-03T14:37:40.099 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000259318 s, 2.0 MB/s 2024-02-03T14:37:40.100 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-03T14:37:40.149 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-02-03T14:37:40.196 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-03T14:37:40.196 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-03T14:37:40.196 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-02-03T14:37:40.196 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-03T14:37:40.197 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-03 14:32:30.964156317 +0000 2024-02-03T14:37:40.197 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-03 14:32:30.672158922 +0000 2024-02-03T14:37:40.197 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-03 14:32:30.672158922 +0000 2024-02-03T14:37:40.197 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-02-03T14:37:40.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-03T14:37:40.250 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-03T14:37:40.250 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-03T14:37:40.250 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000301074 s, 1.7 MB/s 2024-02-03T14:37:40.252 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-03T14:37:40.301 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-03 14:32:31.460151892 +0000 2024-02-03T14:37:40.350 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-03 14:32:31.196154247 +0000 2024-02-03T14:37:40.351 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-03 14:32:31.196154247 +0000 2024-02-03T14:37:40.351 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-02-03T14:37:40.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-03T14:37:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-03T14:37:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-03T14:37:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000308384 s, 1.7 MB/s 2024-02-03T14:37:40.404 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-03T14:37:40.453 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-02-03T14:37:40.500 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-03T14:37:40.500 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-03T14:37:40.500 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-02-03T14:37:40.500 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-03T14:37:40.500 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-02-03 14:32:31.984147217 +0000 2024-02-03T14:37:40.501 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-02-03 14:32:31.716149607 +0000 2024-02-03T14:37:40.501 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-02-03 14:32:31.716149607 +0000 2024-02-03T14:37:40.501 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2024-02-03T14:37:40.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-03T14:37:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-02-03T14:37:40.555 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-02-03T14:37:40.555 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000298324 s, 1.7 MB/s 2024-02-03T14:37:40.556 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-03T14:37:40.605 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-02-03T14:37:40.605 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-03T14:37:40.606 INFO:tasks.ceph:Generating config... 2024-02-03T14:37:40.607 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-02-03T14:37:40.607 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-02-03T14:37:40.607 INFO:tasks.ceph:[global] ms type = async 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-03T14:37:40.607 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-02-03T14:37:40.607 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-03T14:37:40.607 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-02-03T14:37:40.607 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-02-03T14:37:40.607 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-02-03T14:37:40.608 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-02-03T14:37:40.609 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-03T14:37:40.609 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-02-03T14:37:40.609 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-02-03T14:37:40.609 INFO:tasks.ceph:Setting up mon.a... 2024-02-03T14:37:40.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-03T14:37:40.679 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-02-03T14:37:40.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-03T14:37:40.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-03T14:37:40.827 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.7:6789]'), ('mon.b', '[v1:172.21.15.7:6790]'), ('mon.c', '[v1:172.21.15.7:6791]')] 2024-02-03T14:37:40.828 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.7:6789],[v1:172.21.15.7:6790],[v1:172.21.15.7:6791]', 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-02-03T14:37:40.828 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:40.828 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-03T14:37:40.876 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.7:6789]' --addv b '[v1:172.21.15.7:6790]' --addv c '[v1:172.21.15.7:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:40.962 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:40.962 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-02-03T14:37:40.964483+0000 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-02-03T14:37:40.964483+0000 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:0: v1:172.21.15.7:6789/0 mon.a 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:1: v1:172.21.15.7:6790/0 mon.b 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:2: v1:172.21.15.7:6791/0 mon.c 2024-02-03T14:37:40.963 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-03T14:37:40.964 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-03T14:37:41.012 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2dbfabd7-19a5-49d9-9435-d2e80516e779... 2024-02-03T14:37:41.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-03T14:37:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-02-03T14:37:41.089 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-02-03T14:37:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-02-03T14:37:41.091 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-02-03T14:37:41.092 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "[v1:172.21.15.7:6789],[v1:172.21.15.7:6790],[v1:172.21.15.7:6791]" 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 1 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: ms bind msgr2 = False 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.093 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-02-03T14:37:41.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression algorithm = snappy 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression mode = aggressive 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 20 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 20 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2024-02-03T14:37:41.095 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-02-03T14:37:41.096 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.097 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-02-03T14:37:41.098 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2024-02-03T14:37:41.099 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2024-02-03T14:37:41.100 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-03T14:37:41.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-03T14:37:41.192 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-03T14:37:41.192 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:41.193 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-03T14:37:41.199 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:41.199 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-03T14:37:41.244 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:37:41.244 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:41.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-03T14:37:41.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-03T14:37:41.309 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:41.309 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:41.356 INFO:tasks.ceph:Setting up mon nodes... 2024-02-03T14:37:41.357 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-03T14:37:41.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-03T14:37:41.442 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-03T14:37:41.445 INFO:tasks.ceph:Setting up mds nodes... 2024-02-03T14:37:41.445 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-03T14:37:41.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-03T14:37:41.521 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-03T14:37:41.532 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-03T14:37:41.533 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-02-03T14:37:41.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-03T14:37:41.589 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-02-03T14:37:41.589 INFO:tasks.ceph:role: osd.0 2024-02-03T14:37:41.589 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:37:41.589 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-03T14:37:41.645 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-03T14:37:41.645 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-03T14:37:41.645 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-03T14:37:41.646 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-03T14:37:41.669 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-02-03T14:37:41.671 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-02-03T14:37:41.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-03T14:37:41.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-03T14:37:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-02-03T14:37:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:37:41.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-03T14:37:41.844 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-02-03T14:37:41.845 INFO:tasks.ceph:role: osd.1 2024-02-03T14:37:41.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:37:41.845 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-03T14:37:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-03T14:37:41.901 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-03T14:37:41.924 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-02-03T14:37:41.925 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-02-03T14:37:41.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-02-03T14:37:41.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-03T14:37:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:37:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-02-03T14:37:42.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-03T14:37:42.095 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-02-03T14:37:42.095 INFO:tasks.ceph:role: osd.2 2024-02-03T14:37:42.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:37:42.096 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-03T14:37:42.151 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-03T14:37:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-03T14:37:42.151 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-03T14:37:42.152 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-03T14:37:42.175 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-02-03T14:37:42.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-02-03T14:37:42.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-02-03T14:37:42.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-03T14:37:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:37:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: /sbin/restorecon: command not found 2024-02-03T14:37:42.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:42.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-03T14:37:42.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-03T14:37:42.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-03T14:37:42.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-03T14:37:42.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-03T14:37:42.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:42.396+0000 7fddc6c17640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-03T14:37:45.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-03T14:37:45.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-03T14:37:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:45.476+0000 7f990151b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-03T14:37:48.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-03T14:37:48.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.592+0000 7f8b35b07640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-03T14:37:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.592+0000 7f8b35b07640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-03T14:37:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.592+0000 7f8b35b07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-03T14:37:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.592+0000 7f8b35b07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-03T14:37:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.592+0000 7f8b35b07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-03T14:37:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-02-03T14:37:48.596+0000 7f8b35b07640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-03T14:37:51.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-03T14:37:51.613 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-03T14:37:51.613 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:51.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-03T14:37:51.667 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:51.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-03T14:37:51.724 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:51.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-03T14:37:51.779 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:51.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-03T14:37:51.836 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:51.836 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-03T14:37:51.885 INFO:tasks.ceph:Adding keys to all mons... 2024-02-03T14:37:51.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-03T14:37:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-02-03T14:37:51.938 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC1T75lINp/GhAAivNmMlmkLHdbTbi0xZSfag== 2024-02-03T14:37:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-02-03T14:37:51.938 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC2T75l4LjXFxAAUJlAtWONul7R3XuQuJfLLw== 2024-02-03T14:37:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-02-03T14:37:51.939 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC5T75lanuMHBAActgm8MocUpn/yvD4WPNyYg== 2024-02-03T14:37:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-02-03T14:37:51.939 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC8T75l2uKWIxAA/bHZMZz0KSGxoa5YDmE7dA== 2024-02-03T14:37:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-02-03T14:37:51.939 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC1T75l9dgzHxAASJCXGkpTzrd2+8T1SWvvng== 2024-02-03T14:37:51.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-03T14:37:52.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-03T14:37:52.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-03T14:37:52.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-03T14:37:52.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-03T14:37:52.332 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-03T14:37:52.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-03T14:37:52.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-03T14:37:52.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-03T14:37:52.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-03T14:37:52.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-03T14:37:52.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-03T14:37:52.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-03T14:37:52.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-03T14:37:52.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-03T14:37:52.812 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-03T14:37:52.859 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-03T14:37:52.859 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-03T14:37:52.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-02-03T14:37:52.905 INFO:tasks.ceph.mon.a:Started 2024-02-03T14:37:52.905 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-03T14:37:52.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-02-03T14:37:52.907 INFO:tasks.ceph.mon.b:Started 2024-02-03T14:37:52.908 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-03T14:37:52.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-02-03T14:37:52.910 INFO:tasks.ceph.mon.c:Started 2024-02-03T14:37:52.910 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-03T14:37:52.910 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-03T14:37:52.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-02-03T14:37:52.912 INFO:tasks.ceph.mgr.x:Started 2024-02-03T14:37:52.912 INFO:tasks.ceph:Setting crush tunables to default 2024-02-03T14:37:52.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-02-03T14:37:58.429 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-02-03T14:37:58.446 INFO:tasks.ceph:check_enable_crimson: False 2024-02-03T14:37:58.446 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-03T14:37:58.447 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:58.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-03T14:37:58.457 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:58.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-03T14:37:58.512 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:37:58.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-03T14:37:58.536 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:37:58.536+0000 7fd25fcc5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-03T14:37:58.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new 6eff2159-02fa-462e-937d-98e2359ecd93 0 2024-02-03T14:37:58.790 INFO:teuthology.orchestra.run.smithi007.stdout:0 2024-02-03T14:37:58.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new 428cfc4f-5eb3-4048-b602-43dd7d453c2d 1 2024-02-03T14:37:59.002 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-02-03T14:37:59.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new 563c4d0d-9b1e-470d-aaa3-57a4ca92b336 2 2024-02-03T14:37:59.083 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:37:59.080+0000 7fd25fcc5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-03T14:37:59.226 INFO:teuthology.orchestra.run.smithi007.stdout:2 2024-02-03T14:37:59.243 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-03T14:37:59.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-02-03T14:37:59.245 INFO:tasks.ceph.osd.0:Started 2024-02-03T14:37:59.246 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-03T14:37:59.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-02-03T14:37:59.247 INFO:tasks.ceph.osd.1:Started 2024-02-03T14:37:59.248 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-03T14:37:59.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-02-03T14:37:59.249 INFO:tasks.ceph.osd.2:Started 2024-02-03T14:37:59.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-03T14:37:59.333 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:37:59.332+0000 7fd25fcc5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-03T14:37:59.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:37:59.444 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:37:59.225268+0000","last_up_change":"0.000000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:37:59.461 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-03T14:37:59.461 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-03T14:37:59.763 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-03T14:37:59.857 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-02-03T14:37:59.856+0000 7f8098c2d640 -1 Falling back to public interface 2024-02-03T14:37:59.872 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-02-03T14:37:59.872+0000 7fa2346ab640 -1 Falling back to public interface 2024-02-03T14:37:59.876 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-02-03T14:37:59.876+0000 7f80da01b640 -1 Falling back to public interface 2024-02-03T14:37:59.925 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:37:59.924+0000 7fd25fcc5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-03T14:37:59.943 INFO:teuthology.misc.health.smithi007.stdout: 2024-02-03T14:37:59.944 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":5,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:37:59.225268+0000","last_up_change":"0.000000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:37:59.958 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-02-03T14:37:59.982 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:37:59.980+0000 7fd25fcc5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-03T14:38:00.068 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.068+0000 7fd25fcc5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-03T14:38:00.171 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.168+0000 7fd25fcc5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-03T14:38:00.226 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.224+0000 7fd25fcc5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-03T14:38:00.307 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.304+0000 7fd25fcc5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-03T14:38:00.361 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.360+0000 7fd25fcc5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-03T14:38:00.507 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.508+0000 7fd25fcc5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-03T14:38:00.570 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:00.568+0000 7fd25fcc5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-03T14:38:01.222 INFO:tasks.ceph.mgr.x.smithi007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-02-03T14:38:01.223 INFO:tasks.ceph.mgr.x.smithi007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-02-03T14:38:01.223 INFO:tasks.ceph.mgr.x.smithi007.stderr: from numpy import show_config as show_numpy_config 2024-02-03T14:38:01.272 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:01.272+0000 7fd25fcc5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-03T14:38:01.428 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:01.428+0000 7fd25fcc5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-03T14:38:01.600 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:01.600+0000 7fd25fcc5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-03T14:38:01.666 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:01.664+0000 7fd25fcc5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-03T14:38:01.795 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:01.796+0000 7fd25fcc5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-03T14:38:02.409 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:02.408+0000 7fd25fcc5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-03T14:38:02.488 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:02.488+0000 7fd25fcc5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-03T14:38:02.667 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:02.668+0000 7fd25fcc5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-03T14:38:02.744 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:02.744+0000 7fd25fcc5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-03T14:38:03.058 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:03.056+0000 7fd25fcc5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-03T14:38:03.132 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:03.132+0000 7fd25fcc5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-03T14:38:03.742 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-02-03T14:38:03.740+0000 7f8098c2d640 -1 osd.1 0 log_to_monitors true 2024-02-03T14:38:03.749 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-02-03T14:38:03.748+0000 7fa2346ab640 -1 osd.2 0 log_to_monitors true 2024-02-03T14:38:03.766 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-02-03T14:38:03.764+0000 7f80da01b640 -1 osd.0 0 log_to_monitors true 2024-02-03T14:38:04.174 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:04.172+0000 7fd207411640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-03T14:38:04.175 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:38:04.176+0000 7fd207411640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-03T14:38:05.228 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-02-03T14:38:05.228+0000 7fa230654640 -1 osd.2 0 waiting for initial osdmap 2024-02-03T14:38:05.235 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-02-03T14:38:05.236+0000 7fa22c497640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-03T14:38:06.036 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-02-03T14:38:06.036+0000 7f80d67d6640 -1 osd.0 0 waiting for initial osdmap 2024-02-03T14:38:06.039 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-02-03T14:38:06.036+0000 7f80cddff640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-03T14:38:06.041 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-02-03T14:38:06.040+0000 7f8094bd6640 -1 osd.1 0 waiting for initial osdmap 2024-02-03T14:38:06.044 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-02-03T14:38:06.044+0000 7f808ca11640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-03T14:38:06.260 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-03T14:38:06.546 INFO:teuthology.misc.health.smithi007.stdout: 2024-02-03T14:38:06.546 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":8,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:38:06.217130+0000","last_up_change":"2024-02-03T14:38:06.217130+0000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6808","nonce":1938254861}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6809","nonce":1938254861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6811","nonce":1938254861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6810","nonce":1938254861}]},"public_addr":"172.21.15.7:6808/1938254861","cluster_addr":"172.21.15.7:6809/1938254861","heartbeat_back_addr":"172.21.15.7:6811/1938254861","heartbeat_front_addr":"172.21.15.7:6810/1938254861","state":["exists","up"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6800","nonce":2463192558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6801","nonce":2463192558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6803","nonce":2463192558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6802","nonce":2463192558}]},"public_addr":"172.21.15.7:6800/2463192558","cluster_addr":"172.21.15.7:6801/2463192558","heartbeat_back_addr":"172.21.15.7:6803/2463192558","heartbeat_front_addr":"172.21.15.7:6802/2463192558","state":["exists","up"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6804","nonce":4029165148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6805","nonce":4029165148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6807","nonce":4029165148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6806","nonce":4029165148}]},"public_addr":"172.21.15.7:6804/4029165148","cluster_addr":"172.21.15.7:6805/4029165148","heartbeat_back_addr":"172.21.15.7:6807/4029165148","heartbeat_front_addr":"172.21.15.7:6806/4029165148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:38:06.562 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-02-03T14:38:06.563 INFO:tasks.ceph:Creating RBD pool 2024-02-03T14:38:06.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2024-02-03T14:38:07.270 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-02-03T14:38:07.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-03T14:38:08.272 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-03T14:38:08.305 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-03T14:38:08.306 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-03T14:38:08.306 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-03T14:38:08.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-03T14:38:08.306 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-03T14:38:08.649 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:08.664 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6812","nonce":1591058510}]},"active_addr":"172.21.15.7:6812/1591058510","active_change":"2024-02-03T14:38:03.155245+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3870403266}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2903889941}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":493422844}]}]} 2024-02-03T14:38:08.667 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-03T14:38:08.667 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-03T14:38:08.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-03T14:38:08.954 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:08.955 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:38:08.264448+0000","last_up_change":"2024-02-03T14:38:06.217130+0000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-03T14:38:06.885038+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-03T14:38:07.238989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6808","nonce":1938254861}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6809","nonce":1938254861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6811","nonce":1938254861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6810","nonce":1938254861}]},"public_addr":"172.21.15.7:6808/1938254861","cluster_addr":"172.21.15.7:6809/1938254861","heartbeat_back_addr":"172.21.15.7:6811/1938254861","heartbeat_front_addr":"172.21.15.7:6810/1938254861","state":["exists","up"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6800","nonce":2463192558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6801","nonce":2463192558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6803","nonce":2463192558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6802","nonce":2463192558}]},"public_addr":"172.21.15.7:6800/2463192558","cluster_addr":"172.21.15.7:6801/2463192558","heartbeat_back_addr":"172.21.15.7:6803/2463192558","heartbeat_front_addr":"172.21.15.7:6802/2463192558","state":["exists","up"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6804","nonce":4029165148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6805","nonce":4029165148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6807","nonce":4029165148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6806","nonce":4029165148}]},"public_addr":"172.21.15.7:6804/4029165148","cluster_addr":"172.21.15.7:6805/4029165148","heartbeat_back_addr":"172.21.15.7:6807/4029165148","heartbeat_front_addr":"172.21.15.7:6806/4029165148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.753654+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.765229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.713230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:38:08.975 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-03T14:38:08.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-03T14:38:09.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:09.257 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:38:08.264448+0000","last_up_change":"2024-02-03T14:38:06.217130+0000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-03T14:38:06.885038+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-03T14:38:07.238989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6808","nonce":1938254861}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6809","nonce":1938254861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6811","nonce":1938254861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6810","nonce":1938254861}]},"public_addr":"172.21.15.7:6808/1938254861","cluster_addr":"172.21.15.7:6809/1938254861","heartbeat_back_addr":"172.21.15.7:6811/1938254861","heartbeat_front_addr":"172.21.15.7:6810/1938254861","state":["exists","up"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6800","nonce":2463192558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6801","nonce":2463192558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6803","nonce":2463192558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6802","nonce":2463192558}]},"public_addr":"172.21.15.7:6800/2463192558","cluster_addr":"172.21.15.7:6801/2463192558","heartbeat_back_addr":"172.21.15.7:6803/2463192558","heartbeat_front_addr":"172.21.15.7:6802/2463192558","state":["exists","up"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6804","nonce":4029165148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6805","nonce":4029165148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6807","nonce":4029165148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6806","nonce":4029165148}]},"public_addr":"172.21.15.7:6804/4029165148","cluster_addr":"172.21.15.7:6805/4029165148","heartbeat_back_addr":"172.21.15.7:6807/4029165148","heartbeat_front_addr":"172.21.15.7:6806/4029165148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.753654+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.765229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.713230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:38:09.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-03T14:38:09.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-03T14:38:09.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-03T14:38:09.429 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:09.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-03T14:38:09.432 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:09.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-03T14:38:09.433 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:09.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-03T14:38:09.709 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:09.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-02-03T14:38:09.748 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:09.752 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:09.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2024-02-03T14:38:09.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2024-02-03T14:38:10.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-03T14:38:10.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-03T14:38:10.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-03T14:38:11.055 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:11.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2024-02-03T14:38:11.109 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:11.126 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-02-03T14:38:11.134 INFO:teuthology.orchestra.run.smithi007.stdout:34359738369 2024-02-03T14:38:11.150 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2024-02-03T14:38:12.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-03T14:38:12.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-03T14:38:12.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-03T14:38:12.376 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:12.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2024-02-03T14:38:12.393 DEBUG:teuthology.parallel:result is None 2024-02-03T14:38:12.413 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:12.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2024-02-03T14:38:12.430 DEBUG:teuthology.parallel:result is None 2024-02-03T14:38:12.450 INFO:teuthology.orchestra.run.smithi007.stdout:34359738370 2024-02-03T14:38:12.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.2 2024-02-03T14:38:12.466 DEBUG:teuthology.parallel:result is None 2024-02-03T14:38:12.466 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-03T14:38:12.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:38:12.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:12.779 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:38:12.796 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-02-03T14:38:11.163677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89780,"kb_used_data":360,"kb_used_omap":4,"kb_used_meta":89339,"kb_avail":283025740,"statfs":{"total":289910292480,"available":289818357760,"internally_reserved":0,"allocated":368640,"data_stored":873768,"data_compressed":43026,"data_compressed_allocated":57344,"data_compressed_original":778240,"omap_allocated":4785,"internal_metadata":91483471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.272359+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:38:09.272359+0000","last_peered":"2024-02-03T14:38:09.272359+0000","last_clean":"2024-02-03T14:38:09.272359+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:38:09.272359+0000","last_undegraded":"2024-02-03T14:38:09.272359+0000","last_fullsized":"2024-02-03T14:38:09.272359+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.271328+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:38:09.271328+0000","last_peered":"2024-02-03T14:38:09.271328+0000","last_clean":"2024-02-03T14:38:09.271328+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:38:09.271328+0000","last_undegraded":"2024-02-03T14:38:09.271328+0000","last_fullsized":"2024-02-03T14:38:09.271328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270364+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:38:09.270364+0000","last_peered":"2024-02-03T14:38:09.270364+0000","last_clean":"2024-02-03T14:38:09.270364+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:38:09.270364+0000","last_undegraded":"2024-02-03T14:38:09.270364+0000","last_fullsized":"2024-02-03T14:38:09.270364+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270905+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:38:09.270905+0000","last_peered":"2024-02-03T14:38:09.270905+0000","last_clean":"2024-02-03T14:38:09.270905+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:38:09.270905+0000","last_undegraded":"2024-02-03T14:38:09.270905+0000","last_fullsized":"2024-02-03T14:38:09.270905+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270335+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:38:09.270335+0000","last_peered":"2024-02-03T14:38:09.270335+0000","last_clean":"2024-02-03T14:38:09.270335+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:38:09.270335+0000","last_undegraded":"2024-02-03T14:38:09.270335+0000","last_fullsized":"2024-02-03T14:38:09.270335+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.271145+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:38:09.271145+0000","last_peered":"2024-02-03T14:38:09.271145+0000","last_clean":"2024-02-03T14:38:09.271145+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:38:09.271145+0000","last_undegraded":"2024-02-03T14:38:09.271145+0000","last_fullsized":"2024-02-03T14:38:09.271145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-02-03T14:38:08.286117+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:38:08.286117+0000","last_peered":"2024-02-03T14:38:08.286117+0000","last_clean":"2024-02-03T14:38:08.286117+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:38:08.286117+0000","last_undegraded":"2024-02-03T14:38:08.286117+0000","last_fullsized":"2024-02-03T14:38:08.286117+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T20:45:23.868401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.272045+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:38:09.272045+0000","last_peered":"2024-02-03T14:38:09.272045+0000","last_clean":"2024-02-03T14:38:09.272045+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:38:09.272045+0000","last_undegraded":"2024-02-03T14:38:09.272045+0000","last_fullsized":"2024-02-03T14:38:09.272045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270916+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:38:09.270916+0000","last_peered":"2024-02-03T14:38:09.270916+0000","last_clean":"2024-02-03T14:38:09.270916+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:38:09.270916+0000","last_undegraded":"2024-02-03T14:38:09.270916+0000","last_fullsized":"2024-02-03T14:38:09.270916+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":426428,"data_compressed":21513,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":139264,"data_stored":426428,"data_compressed":21513,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31276,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340564,"statfs":{"total":96636764160,"available":96604737536,"internally_reserved":0,"allocated":90112,"data_stored":20912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:38:12.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:38:13.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:13.079 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:38:13.095 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-02-03T14:38:11.163677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89780,"kb_used_data":360,"kb_used_omap":4,"kb_used_meta":89339,"kb_avail":283025740,"statfs":{"total":289910292480,"available":289818357760,"internally_reserved":0,"allocated":368640,"data_stored":873768,"data_compressed":43026,"data_compressed_allocated":57344,"data_compressed_original":778240,"omap_allocated":4785,"internal_metadata":91483471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.272359+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:38:09.272359+0000","last_peered":"2024-02-03T14:38:09.272359+0000","last_clean":"2024-02-03T14:38:09.272359+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:38:09.272359+0000","last_undegraded":"2024-02-03T14:38:09.272359+0000","last_fullsized":"2024-02-03T14:38:09.272359+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.271328+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:38:09.271328+0000","last_peered":"2024-02-03T14:38:09.271328+0000","last_clean":"2024-02-03T14:38:09.271328+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:38:09.271328+0000","last_undegraded":"2024-02-03T14:38:09.271328+0000","last_fullsized":"2024-02-03T14:38:09.271328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270364+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:38:09.270364+0000","last_peered":"2024-02-03T14:38:09.270364+0000","last_clean":"2024-02-03T14:38:09.270364+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:38:09.270364+0000","last_undegraded":"2024-02-03T14:38:09.270364+0000","last_fullsized":"2024-02-03T14:38:09.270364+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270905+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:38:09.270905+0000","last_peered":"2024-02-03T14:38:09.270905+0000","last_clean":"2024-02-03T14:38:09.270905+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:38:09.270905+0000","last_undegraded":"2024-02-03T14:38:09.270905+0000","last_fullsized":"2024-02-03T14:38:09.270905+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270335+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:38:09.270335+0000","last_peered":"2024-02-03T14:38:09.270335+0000","last_clean":"2024-02-03T14:38:09.270335+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:38:09.270335+0000","last_undegraded":"2024-02-03T14:38:09.270335+0000","last_fullsized":"2024-02-03T14:38:09.270335+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.271145+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:38:09.271145+0000","last_peered":"2024-02-03T14:38:09.271145+0000","last_clean":"2024-02-03T14:38:09.271145+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:38:09.271145+0000","last_undegraded":"2024-02-03T14:38:09.271145+0000","last_fullsized":"2024-02-03T14:38:09.271145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-02-03T14:38:08.286117+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:38:08.286117+0000","last_peered":"2024-02-03T14:38:08.286117+0000","last_clean":"2024-02-03T14:38:08.286117+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:38:08.286117+0000","last_undegraded":"2024-02-03T14:38:08.286117+0000","last_fullsized":"2024-02-03T14:38:08.286117+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T20:45:23.868401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.272045+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:38:09.272045+0000","last_peered":"2024-02-03T14:38:09.272045+0000","last_clean":"2024-02-03T14:38:09.272045+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:38:09.272045+0000","last_undegraded":"2024-02-03T14:38:09.272045+0000","last_fullsized":"2024-02-03T14:38:09.272045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-02-03T14:38:09.270916+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:38:09.270916+0000","last_peered":"2024-02-03T14:38:09.270916+0000","last_clean":"2024-02-03T14:38:09.270916+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:38:09.270916+0000","last_undegraded":"2024-02-03T14:38:09.270916+0000","last_fullsized":"2024-02-03T14:38:09.270916+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":139264,"data_stored":426428,"data_compressed":21513,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":139264,"data_stored":426428,"data_compressed":21513,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31276,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340564,"statfs":{"total":96636764160,"available":96604737536,"internally_reserved":0,"allocated":90112,"data_stored":20912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:38:13.096 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-03T14:38:13.096 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-03T14:38:13.096 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-03T14:38:13.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-03T14:38:13.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:38:13.407 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-03T14:38:13.423 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-03T14:38:13.424 INFO:teuthology.run_tasks:Running task workunit... 2024-02-03T14:38:13.437 INFO:tasks.workunit:Pulling workunits from ref 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-03T14:38:13.437 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-03T14:38:13.437 INFO:tasks.workunit:timeout=3h 2024-02-03T14:38:13.437 INFO:tasks.workunit:cleanup=True 2024-02-03T14:38:13.437 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-03T14:38:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:38:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-03T14:38:13.443 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-03T14:38:13.491 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-03T14:38:13.492 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-03T14:38:13.540 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-03T14:38:13.589 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-03T14:39:22.815 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'. 2024-02-03T14:39:22.815 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.815 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-03T14:39:22.815 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-03T14:39:22.815 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:22.816 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at 3cfd0d29cc3 os/bluestore: use blob_offset in compressed read results 2024-02-03T14:39:22.820 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-03T14:39:22.869 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-03T14:39:22.873 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-03T14:39:22.873 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-03T14:39:23.635 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-03T14:39:23.727 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-03T14:39:23.780 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-03T14:39:23.783 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-03T14:39:23.784 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-03T14:39:23.834 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-03T14:39:23.837 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-02-03T14:39:23.837 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-03T14:39:23.888 INFO:tasks.workunit:Running workunits matching cephtool on client.0... 2024-02-03T14:39:23.889 INFO:tasks.workunit:Running workunit cephtool/test.sh... 2024-02-03T14:39:23.890 DEBUG:teuthology.orchestra.run.smithi007:workunit test cephtool/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2024-02-03T14:39:23.941 INFO:tasks.workunit.client.0.smithi007.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ TIMEOUT=300 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ MAX_TIMEOUT=15 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ PG_NUM=4 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ TMPDIR=/tmp 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ type xmlstarlet 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ XMLSTARLET=xmlstarlet 2024-02-03T14:39:23.943 INFO:tasks.workunit.client.0.smithi007.stderr:+++ uname 2024-02-03T14:39:23.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' Linux = FreeBSD ']' 2024-02-03T14:39:23.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ SED=sed 2024-02-03T14:39:23.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ AWK=awk 2024-02-03T14:39:23.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ head -1 2024-02-03T14:39:23.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ stty -a 2024-02-03T14:39:23.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-02-03T14:39:23.957 INFO:tasks.workunit.client.0.smithi007.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-03T14:39:23.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ termwidth= 2024-02-03T14:39:23.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-02-03T14:39:23.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ DIFFCOLOPTS='-y ' 2024-02-03T14:39:23.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ KERNCORE=kernel.core_pattern 2024-02-03T14:39:23.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ EXTRA_OPTS= 2024-02-03T14:39:23.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ test '' = TESTS 2024-02-03T14:39:23.961 INFO:tasks.workunit.client.0.smithi007.stderr:+ set -e 2024-02-03T14:39:23.961 INFO:tasks.workunit.client.0.smithi007.stderr:+ set -o functrace 2024-02-03T14:39:23.962 INFO:tasks.workunit.client.0.smithi007.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-03T14:39:23.962 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:11: : SUDO=sudo 2024-02-03T14:39:23.962 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : export CEPH_DEV=1 2024-02-03T14:39:23.962 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : CEPH_DEV=1 2024-02-03T14:39:23.962 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : mktemp -d /tmp/cephtool.XXX 2024-02-03T14:39:23.963 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : TEMP_DIR=/tmp/cephtool.p0s 2024-02-03T14:39:23.963 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:46: : trap 'rm -fr /tmp/cephtool.p0s' 0 2024-02-03T14:39:23.963 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : mktemp /tmp/cephtool.p0s/test_invalid.XXX 2024-02-03T14:39:23.964 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : TMPFILE=/tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:39:23.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2850: : set +x 2024-02-03T14:39:24.364 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'rbd' already exists 2024-02-03T14:39:25.568 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_injectargs 2024-02-03T14:39:25.568 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:201: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2024-02-03T14:39:25.720 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:25.733 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2024-02-03T14:39:25.734 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: grep false 2024-02-03T14:39:25.887 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_enable_op_tracker": "false" 2024-02-03T14:39:25.888 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:203: test_mon_injectargs: ceph tell osd.0 injectargs '--osd_enable_op_tracker --osd_op_history_duration 500' 2024-02-03T14:39:26.037 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:26.052 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2024-02-03T14:39:26.052 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: grep true 2024-02-03T14:39:26.206 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_enable_op_tracker": "true" 2024-02-03T14:39:26.206 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2024-02-03T14:39:26.207 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: grep 500 2024-02-03T14:39:26.362 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_op_history_duration": "500" 2024-02-03T14:39:26.363 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:206: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2024-02-03T14:39:26.505 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:26.519 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2024-02-03T14:39:26.519 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: grep false 2024-02-03T14:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_enable_op_tracker": "false" 2024-02-03T14:39:26.676 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:208: test_mon_injectargs: ceph tell osd.0 injectargs -- --osd_enable_op_tracker 2024-02-03T14:39:26.826 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:26.841 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2024-02-03T14:39:26.841 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: grep true 2024-02-03T14:39:26.990 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_enable_op_tracker": "true" 2024-02-03T14:39:26.991 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:210: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_enable_op_tracker --osd_op_history_duration 600' 2024-02-03T14:39:27.134 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:27.150 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2024-02-03T14:39:27.150 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: grep true 2024-02-03T14:39:27.310 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_enable_op_tracker": "true" 2024-02-03T14:39:27.310 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2024-02-03T14:39:27.310 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: grep 600 2024-02-03T14:39:27.469 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_op_history_duration": "600" 2024-02-03T14:39:27.469 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:214: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_deep_scrub_interval 2419200' 2024-02-03T14:39:27.615 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:27.629 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: ceph tell osd.0 config get osd_deep_scrub_interval 2024-02-03T14:39:27.629 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: grep 2419200 2024-02-03T14:39:27.782 INFO:tasks.workunit.client.0.smithi007.stdout: "osd_deep_scrub_interval": "2419200.000000" 2024-02-03T14:39:27.782 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:217: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon_probe_timeout 2' 2024-02-03T14:39:27.926 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:27.941 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: ceph tell osd.0 config get mon_probe_timeout 2024-02-03T14:39:27.941 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: grep 2 2024-02-03T14:39:28.103 INFO:tasks.workunit.client.0.smithi007.stdout: "mon_probe_timeout": "2.000000" 2024-02-03T14:39:28.104 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:220: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon-lease 6' 2024-02-03T14:39:28.241 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:28.255 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: ceph tell osd.0 config get mon_lease 2024-02-03T14:39:28.255 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: grep 6 2024-02-03T14:39:28.403 INFO:tasks.workunit.client.0.smithi007.stdout: "mon_lease": "6.000000" 2024-02-03T14:39:28.403 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:224: test_mon_injectargs: expect_false ceph tell osd.0 injectargs --osd-scrub-auto-repair-num-errors -1 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:225: test_mon_injectargs: check_response 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:39:28.537 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:227: test_mon_injectargs: expect_failure /tmp/cephtool.p0s 'Option --osd_op_history_duration requires an argument' ceph tell osd.0 injectargs -- --osd_op_history_duration 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1962: expect_failure: local dir=/tmp/cephtool.p0s 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=Option --osd_op_history_duration requires an argument' 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-02-03T14:39:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 injectargs -- --osd_op_history_duration 2024-02-03T14:39:28.674 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-02-03T14:39:28.674 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1974: expect_failure: false 2024-02-03T14:39:28.674 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'Option --osd_op_history_duration requires an argument' /tmp/cephtool.p0s/out 2024-02-03T14:39:28.674 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-02-03T14:39:28.674 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_injectargs_SI 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.055 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:29.185 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:29.186 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "10000" 2024-02-03T14:39:29.186 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:29.186 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:29.186 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2024-02-03T14:39:29.186 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:29.187 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2024-02-03T14:39:29.188 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2024-02-03T14:39:29.188 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:29.190 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2024-02-03T14:39:29.190 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2024-02-03T14:39:29.190 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:29.190 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: initial_value=10000 2024-02-03T14:39:29.190 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:242: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10 2024-02-03T14:39:29.314 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:39:29.314 INFO:tasks.workunit.client.0.smithi007.stdout: "success": "mon_pg_warn_min_objects = '' (not observed, change may require restart) " 2024-02-03T14:39:29.314 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:39:29.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:243: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2024-02-03T14:39:29.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:29.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:29.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2024-02-03T14:39:29.328 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:29.328 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:29.328 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:29.328 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.329 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:29.458 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:29.459 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "10" 2024-02-03T14:39:29.459 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:29.459 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:29.459 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2024-02-03T14:39:29.459 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:29.461 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2024-02-03T14:39:29.461 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2024-02-03T14:39:29.462 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2024-02-03T14:39:29.463 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:244: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10K 2024-02-03T14:39:29.578 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:39:29.579 INFO:tasks.workunit.client.0.smithi007.stdout: "success": "mon_pg_warn_min_objects = '' (not observed, change may require restart) " 2024-02-03T14:39:29.579 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:245: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.594 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:29.724 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:29.725 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "10000" 2024-02-03T14:39:29.725 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:29.725 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:29.725 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2024-02-03T14:39:29.725 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:29.727 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2024-02-03T14:39:29.727 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2024-02-03T14:39:29.727 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2024-02-03T14:39:29.729 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:246: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 1G 2024-02-03T14:39:29.847 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:39:29.848 INFO:tasks.workunit.client.0.smithi007.stdout: "success": "mon_pg_warn_min_objects = '' (not observed, change may require restart) " 2024-02-03T14:39:29.848 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:39:29.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:247: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2024-02-03T14:39:29.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:29.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:29.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:29.863 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:30.010 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:30.010 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "1000000000" 2024-02-03T14:39:30.010 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:30.011 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:30.011 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2024-02-03T14:39:30.011 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:30.012 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2024-02-03T14:39:30.013 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2024-02-03T14:39:30.013 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:30.014 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2024-02-03T14:39:30.014 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2024-02-03T14:39:30.014 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:30.015 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2024-02-03T14:39:30.015 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2024-02-03T14:39:30.015 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:248: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10F 2024-02-03T14:39:30.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:249: test_mon_injectargs_SI: check_response '(22) Invalid argument' 2024-02-03T14:39:30.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='(22) Invalid argument' 2024-02-03T14:39:30.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:39:30.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:39:30.160 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:39:30.160 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- '(22) Invalid argument' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:39:30.160 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:251: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10' 2024-02-03T14:39:30.301 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:252: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:30.316 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:30.317 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.317 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:30.453 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:30.453 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "10" 2024-02-03T14:39:30.453 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:30.454 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:30.454 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2024-02-03T14:39:30.454 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:30.455 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2024-02-03T14:39:30.456 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2024-02-03T14:39:30.456 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2024-02-03T14:39:30.458 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:253: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10K' 2024-02-03T14:39:30.596 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:30.612 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:254: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:30.613 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:30.614 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:30.614 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.614 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:30.741 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:30.741 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "10000" 2024-02-03T14:39:30.741 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:30.741 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:30.741 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:30.742 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2024-02-03T14:39:30.743 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2024-02-03T14:39:30.743 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2024-02-03T14:39:30.744 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2024-02-03T14:39:30.745 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:255: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 1G' 2024-02-03T14:39:30.876 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:30.891 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:256: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2024-02-03T14:39:30.891 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2024-02-03T14:39:30.891 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2024-02-03T14:39:30.892 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2024-02-03T14:39:31.021 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2024-02-03T14:39:31.021 INFO:tasks.workunit.client.0.smithi007.stderr: "mon_pg_warn_min_objects": "1000000000" 2024-02-03T14:39:31.021 INFO:tasks.workunit.client.0.smithi007.stderr:}' 2024-02-03T14:39:31.021 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2024-02-03T14:39:31.022 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2024-02-03T14:39:31.022 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2024-02-03T14:39:31.023 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2024-02-03T14:39:31.024 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2024-02-03T14:39:31.024 INFO:tasks.workunit.client.0.smithi007.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2024-02-03T14:39:31.025 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2024-02-03T14:39:31.025 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:257: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:31.026 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2024-02-03T14:39:31.153 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: Parse error setting mon_pg_warn_min_objects to '10F' using injectargs (strict_si_cast: unit prefix not recognized). 2024-02-03T14:39:31.154 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:31.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:31.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:258: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2024-02-03T14:39:31.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:31.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2024-02-03T14:39:31.279 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: Parse error setting mon_globalid_prealloc to '-1' using injectargs (strict_sistrtoll: value should not be negative). 2024-02-03T14:39:31.279 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-02-03T14:39:31.281 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:31.281 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:259: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10000 2024-02-03T14:39:31.403 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:39:31.403 INFO:tasks.workunit.client.0.smithi007.stdout: "success": "mon_pg_warn_min_objects = '' (not observed, change may require restart) " 2024-02-03T14:39:31.403 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:39:31.418 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:39:31.808 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_1 2024-02-03T14:39:31.808 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:339: test_tiering_1: ceph osd pool create slow 2 2024-02-03T14:39:32.345 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow' already exists 2024-02-03T14:39:32.360 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:340: test_tiering_1: ceph osd pool application enable slow rados 2024-02-03T14:39:34.268 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'slow' 2024-02-03T14:39:34.295 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:341: test_tiering_1: ceph osd pool create slow2 2 2024-02-03T14:39:35.376 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow2' already exists 2024-02-03T14:39:35.394 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:342: test_tiering_1: ceph osd pool application enable slow2 rados 2024-02-03T14:39:37.294 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'slow2' 2024-02-03T14:39:37.315 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:343: test_tiering_1: ceph osd pool create cache 2 2024-02-03T14:39:38.404 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' already exists 2024-02-03T14:39:38.421 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:344: test_tiering_1: ceph osd pool create cache2 2 2024-02-03T14:39:39.415 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' already exists 2024-02-03T14:39:39.433 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:345: test_tiering_1: ceph osd tier add slow cache 2024-02-03T14:39:40.428 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' is now (or already was) a tier of 'slow' 2024-02-03T14:39:40.444 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:346: test_tiering_1: ceph osd tier add slow cache2 2024-02-03T14:39:41.442 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' is now (or already was) a tier of 'slow' 2024-02-03T14:39:41.459 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:347: test_tiering_1: expect_false ceph osd tier add slow2 cache 2024-02-03T14:39:41.459 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:41.460 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add slow2 cache 2024-02-03T14:39:41.763 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: tier pool 'cache' is already a tier of 'slow' 2024-02-03T14:39:41.767 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:41.768 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: ceph osd pool ls detail -f json 2024-02-03T14:39:41.768 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: grep '{}' 2024-02-03T14:39:41.768 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: jq '.[] | select(.pool_name == "slow") | .application_metadata["rados"]' 2024-02-03T14:39:42.158 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:42.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: ceph osd pool ls detail -f json 2024-02-03T14:39:42.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: jq '.[] | select(.pool_name == "slow2") | .application_metadata["rados"]' 2024-02-03T14:39:42.159 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: grep '{}' 2024-02-03T14:39:42.557 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:42.558 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: ceph osd pool ls detail -f json 2024-02-03T14:39:42.558 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: grep '{}' 2024-02-03T14:39:42.558 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: jq '.[] | select(.pool_name == "cache") | .application_metadata["rados"]' 2024-02-03T14:39:42.963 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:42.964 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: ceph osd pool ls detail -f json 2024-02-03T14:39:42.964 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: jq '.[] | select(.pool_name == "cache2") | .application_metadata["rados"]' 2024-02-03T14:39:42.964 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: grep '{}' 2024-02-03T14:39:43.356 INFO:tasks.workunit.client.0.smithi007.stdout:{} 2024-02-03T14:39:43.357 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:354: test_tiering_1: expect_false ceph osd tier cache-mode cache forward 2024-02-03T14:39:43.357 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:43.357 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward 2024-02-03T14:39:43.658 INFO:tasks.workunit.client.0.smithi007.stderr:Invalid command: forward not in writeback|proxy|readproxy|readonly|none 2024-02-03T14:39:43.658 INFO:tasks.workunit.client.0.smithi007.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2024-02-03T14:39:43.658 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: invalid command 2024-02-03T14:39:43.661 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:43.661 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:355: test_tiering_1: expect_false ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2024-02-03T14:39:43.661 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:43.661 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2024-02-03T14:39:43.968 INFO:tasks.workunit.client.0.smithi007.stderr:Invalid command: forward not in writeback|proxy|readproxy|readonly|none 2024-02-03T14:39:43.968 INFO:tasks.workunit.client.0.smithi007.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2024-02-03T14:39:43.968 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: invalid command 2024-02-03T14:39:43.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:43.972 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:357: test_tiering_1: ceph osd tier cache-mode cache writeback 2024-02-03T14:39:45.126 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to writeback 2024-02-03T14:39:45.145 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:358: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly 2024-02-03T14:39:45.145 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:45.145 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly 2024-02-03T14:39:45.439 INFO:tasks.workunit.client.0.smithi007.stderr:Error EPERM: 'readonly' is not a well-supported cache mode and may corrupt your data. pass --yes-i-really-mean-it to force. 2024-02-03T14:39:45.443 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:45.443 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:359: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:45.443 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:45.443 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:45.746 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: unable to set cache-mode 'readonly' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2024-02-03T14:39:45.752 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:45.752 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:360: test_tiering_1: ceph osd tier cache-mode cache proxy 2024-02-03T14:39:47.125 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to proxy 2024-02-03T14:39:47.142 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:361: test_tiering_1: ceph osd tier cache-mode cache readproxy 2024-02-03T14:39:48.469 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to readproxy 2024-02-03T14:39:48.486 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:362: test_tiering_1: ceph osd tier cache-mode cache none 2024-02-03T14:39:49.474 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to none 2024-02-03T14:39:49.492 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:363: test_tiering_1: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:50.478 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to readonly 2024-02-03T14:39:50.496 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:364: test_tiering_1: ceph osd tier cache-mode cache none 2024-02-03T14:39:51.479 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to none 2024-02-03T14:39:51.496 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:365: test_tiering_1: ceph osd tier cache-mode cache writeback 2024-02-03T14:39:52.483 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to writeback 2024-02-03T14:39:52.500 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:366: test_tiering_1: ceph osd tier cache-mode cache proxy 2024-02-03T14:39:53.499 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to proxy 2024-02-03T14:39:53.514 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:367: test_tiering_1: ceph osd tier cache-mode cache writeback 2024-02-03T14:39:54.510 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to writeback 2024-02-03T14:39:54.527 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:368: test_tiering_1: expect_false ceph osd tier cache-mode cache none 2024-02-03T14:39:54.527 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:54.528 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache none 2024-02-03T14:39:54.835 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: unable to set cache-mode 'none' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2024-02-03T14:39:54.840 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:54.840 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:369: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:54.840 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:54.840 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:55.145 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: unable to set cache-mode 'readonly' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2024-02-03T14:39:55.151 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:55.151 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:372: test_tiering_1: rados -p cache put /etc/passwd /etc/passwd 2024-02-03T14:39:55.193 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:373: test_tiering_1: flush_pg_stats 2024-02-03T14:39:55.193 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-03T14:39:55.193 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-03T14:39:55.574 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-03T14:39:55.574 INFO:tasks.workunit.client.0.smithi007.stderr:1 2024-02-03T14:39:55.574 INFO:tasks.workunit.client.0.smithi007.stderr:2' 2024-02-03T14:39:55.574 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-03T14:39:55.574 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:39:55.575 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-03T14:39:55.742 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738393 2024-02-03T14:39:55.742 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738393 2024-02-03T14:39:55.742 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738393' 2024-02-03T14:39:55.743 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:39:55.743 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-03T14:39:55.893 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738393 2024-02-03T14:39:55.893 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738393 2024-02-03T14:39:55.893 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738393 1-34359738393' 2024-02-03T14:39:55.893 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:39:55.893 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-03T14:39:56.041 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738393 2024-02-03T14:39:56.042 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738393 2024-02-03T14:39:56.042 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738393 1-34359738393 2-34359738393' 2024-02-03T14:39:56.042 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:39:56.042 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-34359738393 2024-02-03T14:39:56.042 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:39:56.043 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-03T14:39:56.044 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-34359738393 2024-02-03T14:39:56.044 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:39:56.045 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.0 seq 34359738393 2024-02-03T14:39:56.045 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738393 2024-02-03T14:39:56.045 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 34359738393' 2024-02-03T14:39:56.045 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:39:56.446 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738391 -lt 34359738393 2024-02-03T14:39:56.446 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-03T14:39:57.447 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-03T14:39:57.448 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:39:57.854 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738394 -lt 34359738393 2024-02-03T14:39:57.854 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:39:57.855 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-34359738393 2024-02-03T14:39:57.855 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:39:57.856 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-03T14:39:57.857 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-34359738393 2024-02-03T14:39:57.857 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:39:57.858 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738393 2024-02-03T14:39:57.858 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.1 seq 34359738393 2024-02-03T14:39:57.859 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 34359738393' 2024-02-03T14:39:57.859 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-03T14:39:58.260 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738394 -lt 34359738393 2024-02-03T14:39:58.260 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:39:58.260 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-34359738393 2024-02-03T14:39:58.260 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:39:58.261 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-03T14:39:58.262 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-34359738393 2024-02-03T14:39:58.262 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:39:58.263 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738393 2024-02-03T14:39:58.264 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 34359738393' 2024-02-03T14:39:58.264 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.2 seq 34359738393 2024-02-03T14:39:58.264 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-03T14:39:58.672 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738394 -lt 34359738393 2024-02-03T14:39:58.672 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:375: test_tiering_1: ceph osd tier cache-mode cache proxy 2024-02-03T14:39:59.530 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to proxy 2024-02-03T14:39:59.547 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:376: test_tiering_1: expect_false ceph osd tier cache-mode cache none 2024-02-03T14:39:59.547 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:59.548 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache none 2024-02-03T14:39:59.842 INFO:tasks.workunit.client.0.smithi007.stderr:Error EBUSY: unable to set cache-mode 'none' on pool 'cache': dirty objects found 2024-02-03T14:39:59.846 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:39:59.847 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:377: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:39:59.847 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:39:59.847 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:40:00.154 INFO:tasks.workunit.client.0.smithi007.stderr:Error EBUSY: unable to set cache-mode 'readonly' on pool 'cache': dirty objects found 2024-02-03T14:40:00.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:00.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:378: test_tiering_1: ceph osd tier cache-mode cache writeback 2024-02-03T14:40:01.545 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to writeback 2024-02-03T14:40:01.562 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:380: test_tiering_1: rados -p cache rm /etc/passwd 2024-02-03T14:40:01.603 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:381: test_tiering_1: rados -p cache cache-flush-evict-all 2024-02-03T14:40:01.631 INFO:tasks.workunit.client.0.smithi007.stdout: /etc/passwd 2024-02-03T14:40:01.636 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:382: test_tiering_1: flush_pg_stats 2024-02-03T14:40:01.637 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-03T14:40:01.637 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr:1 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr:2' 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:40:02.011 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-03T14:40:02.177 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738397 2024-02-03T14:40:02.177 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738397 2024-02-03T14:40:02.177 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738397' 2024-02-03T14:40:02.177 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:40:02.177 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-03T14:40:02.334 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738397 2024-02-03T14:40:02.335 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738397 2024-02-03T14:40:02.335 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738397 1-34359738397' 2024-02-03T14:40:02.335 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:40:02.335 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738397 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738397 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738397 1-34359738397 2-34359738397' 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-34359738397 2024-02-03T14:40:02.494 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:40:02.495 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-03T14:40:02.495 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-34359738397 2024-02-03T14:40:02.496 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:40:02.496 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738397 2024-02-03T14:40:02.497 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.0 seq 34359738397 2024-02-03T14:40:02.497 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 34359738397' 2024-02-03T14:40:02.497 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:40:02.887 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738394 -lt 34359738397 2024-02-03T14:40:02.888 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-03T14:40:03.889 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-03T14:40:03.890 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:40:04.288 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738397 -lt 34359738397 2024-02-03T14:40:04.288 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:40:04.289 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-34359738397 2024-02-03T14:40:04.289 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:40:04.291 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-03T14:40:04.291 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-34359738397 2024-02-03T14:40:04.291 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:40:04.292 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738397 2024-02-03T14:40:04.293 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.1 seq 34359738397 2024-02-03T14:40:04.293 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 34359738397' 2024-02-03T14:40:04.293 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-03T14:40:04.678 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738397 -lt 34359738397 2024-02-03T14:40:04.678 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:40:04.679 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:40:04.679 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-34359738397 2024-02-03T14:40:04.679 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-03T14:40:04.680 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-34359738397 2024-02-03T14:40:04.680 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:40:04.681 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738397 2024-02-03T14:40:04.682 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.2 seq 34359738397 2024-02-03T14:40:04.682 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 34359738397' 2024-02-03T14:40:04.682 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-03T14:40:05.066 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738397 -lt 34359738397 2024-02-03T14:40:05.067 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:384: test_tiering_1: ceph osd tier cache-mode cache proxy 2024-02-03T14:40:06.139 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to proxy 2024-02-03T14:40:06.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:385: test_tiering_1: ceph osd tier cache-mode cache none 2024-02-03T14:40:07.149 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to none 2024-02-03T14:40:07.167 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:386: test_tiering_1: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2024-02-03T14:40:08.160 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to readonly 2024-02-03T14:40:08.178 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:387: test_tiering_1: TRIES=0 2024-02-03T14:40:08.178 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:388: test_tiering_1: ceph osd pool set cache pg_num 3 --yes-i-really-mean-it 2024-02-03T14:40:10.089 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:395: test_tiering_1: expect_false ceph osd pool set cache pg_num 4 2024-02-03T14:40:10.090 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:10.090 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd pool set cache pg_num 4 2024-02-03T14:40:10.382 INFO:tasks.workunit.client.0.smithi007.stderr:Error EPERM: splits in cache pools must be followed by scrubs and leave sufficient free space to avoid overfilling. use --yes-i-really-mean-it to force. 2024-02-03T14:40:10.388 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:10.388 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:396: test_tiering_1: ceph osd tier cache-mode cache none 2024-02-03T14:40:11.185 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache' to none 2024-02-03T14:40:11.203 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:397: test_tiering_1: ceph osd tier set-overlay slow cache 2024-02-03T14:40:12.201 INFO:tasks.workunit.client.0.smithi007.stderr:overlay for 'slow' is now (or already was) 'cache' 2024-02-03T14:40:12.218 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:398: test_tiering_1: expect_false ceph osd tier set-overlay slow cache2 2024-02-03T14:40:12.219 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:12.219 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier set-overlay slow cache2 2024-02-03T14:40:12.525 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: pool 'slow' has overlay 'cache'; please remove-overlay first 2024-02-03T14:40:12.530 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:12.531 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:399: test_tiering_1: expect_false ceph osd tier remove slow cache 2024-02-03T14:40:12.531 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:12.531 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier remove slow cache 2024-02-03T14:40:12.838 INFO:tasks.workunit.client.0.smithi007.stderr:Error EBUSY: tier pool 'cache' is the overlay for 'slow'; please remove-overlay first 2024-02-03T14:40:12.844 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:12.844 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:400: test_tiering_1: ceph osd tier remove-overlay slow 2024-02-03T14:40:14.207 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'slow' 2024-02-03T14:40:14.224 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:401: test_tiering_1: ceph osd tier set-overlay slow cache2 2024-02-03T14:40:15.216 INFO:tasks.workunit.client.0.smithi007.stderr:overlay for 'slow' is now (or already was) 'cache2' 2024-02-03T14:40:15.234 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:402: test_tiering_1: ceph osd tier remove-overlay slow 2024-02-03T14:40:16.218 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'slow' 2024-02-03T14:40:16.236 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:403: test_tiering_1: ceph osd tier remove slow cache 2024-02-03T14:40:17.239 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' is now (or already was) not a tier of 'slow' 2024-02-03T14:40:17.255 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:404: test_tiering_1: ceph osd tier add slow2 cache 2024-02-03T14:40:18.239 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' is now (or already was) a tier of 'slow2' 2024-02-03T14:40:18.257 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:405: test_tiering_1: expect_false ceph osd tier set-overlay slow cache 2024-02-03T14:40:18.257 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:18.257 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier set-overlay slow cache 2024-02-03T14:40:18.562 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: tier pool 'cache' is not a tier of 'slow' 2024-02-03T14:40:18.568 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:18.568 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:406: test_tiering_1: ceph osd tier set-overlay slow2 cache 2024-02-03T14:40:19.250 INFO:tasks.workunit.client.0.smithi007.stderr:overlay for 'slow2' is now (or already was) 'cache' 2024-02-03T14:40:19.268 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:407: test_tiering_1: ceph osd tier remove-overlay slow2 2024-02-03T14:40:20.269 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'slow2' 2024-02-03T14:40:20.286 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:408: test_tiering_1: ceph osd tier remove slow2 cache 2024-02-03T14:40:21.272 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' is now (or already was) not a tier of 'slow2' 2024-02-03T14:40:21.290 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:409: test_tiering_1: ceph osd tier remove slow cache2 2024-02-03T14:40:22.279 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' is now (or already was) not a tier of 'slow' 2024-02-03T14:40:22.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:412: test_tiering_1: rados -p cache2 put /etc/passwd /etc/passwd 2024-02-03T14:40:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2024-02-03T14:40:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2024-02-03T14:40:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2024-02-03T14:40:22.802 INFO:tasks.workunit.client.0.smithi007.stdout:waiting for pg stats to flush 2024-02-03T14:40:22.802 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:414: test_tiering_1: echo waiting for pg stats to flush 2024-02-03T14:40:22.802 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:415: test_tiering_1: sleep 2 2024-02-03T14:40:24.805 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2024-02-03T14:40:24.805 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2024-02-03T14:40:24.805 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2024-02-03T14:40:25.313 INFO:tasks.workunit.client.0.smithi007.stdout:waiting for pg stats to flush 2024-02-03T14:40:25.314 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:414: test_tiering_1: echo waiting for pg stats to flush 2024-02-03T14:40:25.314 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:415: test_tiering_1: sleep 2 2024-02-03T14:40:27.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2024-02-03T14:40:27.316 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2024-02-03T14:40:27.317 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2024-02-03T14:40:27.820 INFO:tasks.workunit.client.0.smithi007.stdout:cache2 6 2 11 KiB 1 16 KiB 0 128 GiB 2024-02-03T14:40:27.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:417: test_tiering_1: expect_false ceph osd tier add slow cache2 2024-02-03T14:40:27.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:27.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add slow cache2 2024-02-03T14:40:28.117 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOTEMPTY: tier pool 'cache2' is not empty; --force-nonempty to force 2024-02-03T14:40:28.121 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:28.122 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:418: test_tiering_1: ceph osd tier add slow cache2 --force-nonempty 2024-02-03T14:40:29.377 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' is now (or already was) a tier of 'slow' 2024-02-03T14:40:29.394 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:419: test_tiering_1: ceph osd tier remove slow cache2 2024-02-03T14:40:30.379 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' is now (or already was) not a tier of 'slow' 2024-02-03T14:40:30.396 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:421: test_tiering_1: grep cache2 2024-02-03T14:40:30.396 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:421: test_tiering_1: ceph osd pool ls 2024-02-03T14:40:30.784 INFO:tasks.workunit.client.0.smithi007.stdout:cache2 2024-02-03T14:40:30.784 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:422: test_tiering_1: ceph osd pool ls -f json-pretty 2024-02-03T14:40:30.784 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:422: test_tiering_1: grep cache2 2024-02-03T14:40:31.183 INFO:tasks.workunit.client.0.smithi007.stdout: "cache2" 2024-02-03T14:40:31.183 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:423: test_tiering_1: ceph osd pool ls detail 2024-02-03T14:40:31.183 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:423: test_tiering_1: grep cache2 2024-02-03T14:40:31.577 INFO:tasks.workunit.client.0.smithi007.stdout:pool 6 'cache2' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 50 lfor 42/42/42 flags hashpspool stripe_width 0 application rados read_balance_score 1.50 2024-02-03T14:40:31.577 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:424: test_tiering_1: ceph osd pool ls detail -f json-pretty 2024-02-03T14:40:31.577 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:424: test_tiering_1: grep cache2 2024-02-03T14:40:31.970 INFO:tasks.workunit.client.0.smithi007.stdout: "pool_name": "cache2", 2024-02-03T14:40:31.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:426: test_tiering_1: ceph osd pool delete slow slow --yes-i-really-really-mean-it 2024-02-03T14:40:32.389 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow' does not exist 2024-02-03T14:40:32.406 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:427: test_tiering_1: ceph osd pool delete slow2 slow2 --yes-i-really-really-mean-it 2024-02-03T14:40:33.394 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow2' does not exist 2024-02-03T14:40:33.412 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:428: test_tiering_1: ceph osd pool delete cache cache --yes-i-really-really-mean-it 2024-02-03T14:40:34.403 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache' does not exist 2024-02-03T14:40:34.421 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:429: test_tiering_1: ceph osd pool delete cache2 cache2 --yes-i-really-really-mean-it 2024-02-03T14:40:35.419 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache2' does not exist 2024-02-03T14:40:35.438 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:40:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_2 2024-02-03T14:40:35.829 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:435: test_tiering_2: ceph osd pool create snap_base 2 2024-02-03T14:40:36.422 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'snap_base' already exists 2024-02-03T14:40:36.439 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:436: test_tiering_2: ceph osd pool application enable snap_base rados 2024-02-03T14:40:38.338 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'snap_base' 2024-02-03T14:40:38.359 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:437: test_tiering_2: ceph osd pool create snap_cache 2 2024-02-03T14:40:39.453 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'snap_cache' already exists 2024-02-03T14:40:39.472 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:438: test_tiering_2: ceph osd pool mksnap snap_cache snapname 2024-02-03T14:40:40.454 INFO:tasks.workunit.client.0.smithi007.stderr:pool snap_cache snap snapname already exists 2024-02-03T14:40:40.472 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:439: test_tiering_2: expect_false ceph osd tier add snap_base snap_cache 2024-02-03T14:40:40.472 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:40.472 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add snap_base snap_cache 2024-02-03T14:40:40.777 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOTEMPTY: tier pool 'snap_cache' has snapshot state; it cannot be added as a tier without breaking the pool 2024-02-03T14:40:40.783 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:40.783 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:440: test_tiering_2: ceph osd pool delete snap_base snap_base --yes-i-really-really-mean-it 2024-02-03T14:40:41.466 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'snap_base' does not exist 2024-02-03T14:40:41.484 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:441: test_tiering_2: ceph osd pool delete snap_cache snap_cache --yes-i-really-really-mean-it 2024-02-03T14:40:42.468 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'snap_cache' does not exist 2024-02-03T14:40:42.484 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:40:42.888 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_3 2024-02-03T14:40:42.888 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:447: test_tiering_3: ceph osd pool create basex 2 2024-02-03T14:40:44.136 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basex' already exists 2024-02-03T14:40:44.153 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:448: test_tiering_3: ceph osd pool application enable basex rados 2024-02-03T14:40:46.046 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'basex' 2024-02-03T14:40:46.069 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:449: test_tiering_3: ceph osd pool create cachex 2 2024-02-03T14:40:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachex' already exists 2024-02-03T14:40:47.169 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:450: test_tiering_3: ceph osd tier add basex cachex 2024-02-03T14:40:48.178 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachex' is now (or already was) a tier of 'basex' 2024-02-03T14:40:48.196 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:451: test_tiering_3: expect_false ceph osd pool mksnap cache snapname 2024-02-03T14:40:48.196 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:48.196 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd pool mksnap cache snapname 2024-02-03T14:40:48.483 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOENT: unrecognized pool 'cache' 2024-02-03T14:40:48.487 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:48.488 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:452: test_tiering_3: ceph osd tier remove basex cachex 2024-02-03T14:40:49.178 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachex' is now (or already was) not a tier of 'basex' 2024-02-03T14:40:49.195 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:453: test_tiering_3: ceph osd pool delete basex basex --yes-i-really-really-mean-it 2024-02-03T14:40:50.178 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basex' does not exist 2024-02-03T14:40:50.195 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:454: test_tiering_3: ceph osd pool delete cachex cachex --yes-i-really-really-mean-it 2024-02-03T14:40:51.187 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachex' does not exist 2024-02-03T14:40:51.205 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:40:51.605 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_4 2024-02-03T14:40:51.606 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:460: test_tiering_4: ceph osd pool create eccache 2 2 erasure 2024-02-03T14:40:53.203 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'eccache' already exists 2024-02-03T14:40:53.220 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:461: test_tiering_4: expect_false ceph osd set-require-min-compat-client bobtail 2024-02-03T14:40:53.221 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:53.221 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd set-require-min-compat-client bobtail 2024-02-03T14:40:53.516 INFO:tasks.workunit.client.0.smithi007.stderr:Error EPERM: osdmap current utilizes features that require jewel; cannot set require_min_compat_client below that to bobtail 2024-02-03T14:40:53.520 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:53.520 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:462: test_tiering_4: ceph osd pool create repbase 2 2024-02-03T14:40:54.211 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'repbase' already exists 2024-02-03T14:40:54.228 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:463: test_tiering_4: ceph osd pool application enable repbase rados 2024-02-03T14:40:56.131 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'repbase' 2024-02-03T14:40:56.152 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:464: test_tiering_4: expect_false ceph osd tier add repbase eccache 2024-02-03T14:40:56.152 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:40:56.152 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add repbase eccache 2024-02-03T14:40:56.460 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOTSUP: tier pool 'eccache' is an ec pool, which cannot be a tier 2024-02-03T14:40:56.465 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:40:56.466 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:465: test_tiering_4: ceph osd pool delete repbase repbase --yes-i-really-really-mean-it 2024-02-03T14:40:57.234 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'repbase' does not exist 2024-02-03T14:40:57.252 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:466: test_tiering_4: ceph osd pool delete eccache eccache --yes-i-really-really-mean-it 2024-02-03T14:40:58.246 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'eccache' does not exist 2024-02-03T14:40:58.264 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:40:58.643 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_5 2024-02-03T14:40:58.644 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:472: test_tiering_5: ceph osd pool create slow 2 2024-02-03T14:40:59.247 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow' already exists 2024-02-03T14:40:59.264 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:473: test_tiering_5: ceph osd pool application enable slow rados 2024-02-03T14:41:01.166 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'slow' 2024-02-03T14:41:01.194 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:474: test_tiering_5: ceph osd pool create cache3 2 2024-02-03T14:41:02.290 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache3' already exists 2024-02-03T14:41:02.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:475: test_tiering_5: ceph osd tier add-cache slow cache3 1024000 2024-02-03T14:41:03.305 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache3' is now (or already was) a tier of 'slow' 2024-02-03T14:41:03.322 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: ceph osd dump 2024-02-03T14:41:03.322 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep cache3 2024-02-03T14:41:03.322 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep bloom 2024-02-03T14:41:03.322 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep 'false_positive_probability: 0.05' 2024-02-03T14:41:03.322 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep 'target_bytes 1024000' 2024-02-03T14:41:03.323 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep '1200s x4' 2024-02-03T14:41:03.720 INFO:tasks.workunit.client.0.smithi007.stdout:pool 14 'cache3' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 81 lfor 81/81/81 flags hashpspool,creating tier_of 13 cache_mode writeback target_bytes 1024000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 1200s x4 decay_rate 20 search_last_n 1 min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0 application rados read_balance_score 1.50 2024-02-03T14:41:03.720 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:477: test_tiering_5: ceph osd tier remove slow cache3 2024-02-03T14:41:04.031 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:477: test_tiering_5: true 2024-02-03T14:41:04.031 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:478: test_tiering_5: check_response 'EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:41:04.032 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:479: test_tiering_5: ceph osd tier remove-overlay slow 2024-02-03T14:41:05.318 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'slow' 2024-02-03T14:41:05.335 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:480: test_tiering_5: ceph osd tier remove slow cache3 2024-02-03T14:41:06.326 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache3' is now (or already was) not a tier of 'slow' 2024-02-03T14:41:06.343 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:481: test_tiering_5: ceph osd pool ls 2024-02-03T14:41:06.343 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:481: test_tiering_5: grep cache3 2024-02-03T14:41:06.737 INFO:tasks.workunit.client.0.smithi007.stdout:cache3 2024-02-03T14:41:06.737 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:482: test_tiering_5: ceph osd pool delete cache3 cache3 --yes-i-really-really-mean-it 2024-02-03T14:41:07.309 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache3' does not exist 2024-02-03T14:41:07.325 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:483: test_tiering_5: ceph osd pool ls 2024-02-03T14:41:07.325 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:483: test_tiering_5: grep cache3 2024-02-03T14:41:07.720 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:484: test_tiering_5: ceph osd pool delete slow slow --yes-i-really-really-mean-it 2024-02-03T14:41:08.368 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'slow' does not exist 2024-02-03T14:41:08.386 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:41:08.784 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_6 2024-02-03T14:41:08.784 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:490: test_tiering_6: ceph osd pool create datapool 2 2024-02-03T14:41:09.374 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' already exists 2024-02-03T14:41:09.391 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:491: test_tiering_6: ceph osd pool application enable datapool rados 2024-02-03T14:41:11.287 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'datapool' 2024-02-03T14:41:11.309 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:492: test_tiering_6: ceph osd pool create cachepool 2 2024-02-03T14:41:12.407 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' already exists 2024-02-03T14:41:12.424 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:493: test_tiering_6: ceph osd tier add-cache datapool cachepool 1024000 2024-02-03T14:41:13.427 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' is now (or already was) a tier of 'datapool' 2024-02-03T14:41:13.444 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:494: test_tiering_6: ceph osd tier cache-mode cachepool writeback 2024-02-03T14:41:13.859 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cachepool' to writeback 2024-02-03T14:41:13.876 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:495: test_tiering_6: rados -p datapool put object /etc/passwd 2024-02-03T14:41:13.919 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:496: test_tiering_6: rados -p cachepool stat object 2024-02-03T14:41:13.950 INFO:tasks.workunit.client.0.smithi007.stdout:cachepool/object mtime 2024-02-03T14:41:13.000000+0000, size 11677 2024-02-03T14:41:13.952 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:497: test_tiering_6: rados -p cachepool cache-flush object 2024-02-03T14:41:13.985 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:498: test_tiering_6: rados -p datapool stat object 2024-02-03T14:41:14.014 INFO:tasks.workunit.client.0.smithi007.stdout:datapool/object mtime 2024-02-03T14:41:13.000000+0000, size 11677 2024-02-03T14:41:14.016 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:499: test_tiering_6: ceph osd tier remove-overlay datapool 2024-02-03T14:41:15.426 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'datapool' 2024-02-03T14:41:15.443 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:500: test_tiering_6: ceph osd tier remove datapool cachepool 2024-02-03T14:41:16.434 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' is now (or already was) not a tier of 'datapool' 2024-02-03T14:41:16.452 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:501: test_tiering_6: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2024-02-03T14:41:17.428 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' does not exist 2024-02-03T14:41:17.445 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:502: test_tiering_6: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2024-02-03T14:41:18.431 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' does not exist 2024-02-03T14:41:18.449 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:41:18.869 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_7 2024-02-03T14:41:18.869 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:508: test_tiering_7: ceph osd pool create datapool 2 2024-02-03T14:41:19.445 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' already exists 2024-02-03T14:41:19.462 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:509: test_tiering_7: ceph osd pool application enable datapool rados 2024-02-03T14:41:21.371 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'datapool' 2024-02-03T14:41:21.392 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:510: test_tiering_7: ceph osd pool create cachepool 2 2024-02-03T14:41:22.476 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' already exists 2024-02-03T14:41:22.493 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:511: test_tiering_7: ceph osd tier add-cache datapool cachepool 1024000 2024-02-03T14:41:23.507 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' is now (or already was) a tier of 'datapool' 2024-02-03T14:41:23.525 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:512: test_tiering_7: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2024-02-03T14:41:23.820 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:512: test_tiering_7: true 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:513: test_tiering_7: check_response 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:41:23.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:41:23.822 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:514: test_tiering_7: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:514: test_tiering_7: true 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:515: test_tiering_7: check_response 'EBUSY: pool '\''datapool'\'' has tiers cachepool' 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: pool '\''datapool'\'' has tiers cachepool' 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:41:24.129 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: pool '\''datapool'\'' has tiers cachepool' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:41:24.130 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:516: test_tiering_7: ceph osd tier remove-overlay datapool 2024-02-03T14:41:25.519 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'datapool' 2024-02-03T14:41:25.536 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:517: test_tiering_7: ceph osd tier remove datapool cachepool 2024-02-03T14:41:26.536 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' is now (or already was) not a tier of 'datapool' 2024-02-03T14:41:26.553 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:518: test_tiering_7: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2024-02-03T14:41:27.527 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cachepool' does not exist 2024-02-03T14:41:27.544 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:519: test_tiering_7: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2024-02-03T14:41:28.538 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' does not exist 2024-02-03T14:41:28.555 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:41:28.942 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_8 2024-02-03T14:41:28.942 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:525: test_tiering_8: ceph osd set notieragent 2024-02-03T14:41:30.459 INFO:tasks.workunit.client.0.smithi007.stderr:notieragent is set 2024-02-03T14:41:30.482 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:526: test_tiering_8: ceph osd pool create datapool 2 2024-02-03T14:41:31.563 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' already exists 2024-02-03T14:41:31.581 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:527: test_tiering_8: ceph osd pool application enable datapool rados 2024-02-03T14:41:33.475 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'datapool' 2024-02-03T14:41:33.500 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:528: test_tiering_8: ceph osd pool create cache4 2 2024-02-03T14:41:34.598 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache4' already exists 2024-02-03T14:41:34.615 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:529: test_tiering_8: ceph osd tier add-cache datapool cache4 1024000 2024-02-03T14:41:35.621 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache4' is now (or already was) a tier of 'datapool' 2024-02-03T14:41:35.639 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:530: test_tiering_8: ceph osd tier cache-mode cache4 writeback 2024-02-03T14:41:36.055 INFO:tasks.workunit.client.0.smithi007.stderr:set cache-mode for pool 'cache4' to writeback 2024-02-03T14:41:36.072 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: mktemp 2024-02-03T14:41:36.072 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: grep tmp 2024-02-03T14:41:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: tmpfile=/tmp/tmp.9eMPrUw89T 2024-02-03T14:41:36.074 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:532: test_tiering_8: dd if=/dev/zero of=/tmp/tmp.9eMPrUw89T bs=4K count=1 2024-02-03T14:41:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:1+0 records in 2024-02-03T14:41:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:1+0 records out 2024-02-03T14:41:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000142233 s, 28.8 MB/s 2024-02-03T14:41:36.075 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:533: test_tiering_8: ceph osd pool set cache4 target_max_objects 200 2024-02-03T14:41:37.524 INFO:tasks.workunit.client.0.smithi007.stderr:set pool 20 target_max_objects to 200 2024-02-03T14:41:37.545 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:534: test_tiering_8: ceph osd pool set cache4 target_max_bytes 1000000 2024-02-03T14:41:39.528 INFO:tasks.workunit.client.0.smithi007.stderr:set pool 20 target_max_bytes to 1000000 2024-02-03T14:41:39.549 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:535: test_tiering_8: rados -p cache4 put foo1 /tmp/tmp.9eMPrUw89T 2024-02-03T14:41:39.587 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:536: test_tiering_8: rados -p cache4 put foo2 /tmp/tmp.9eMPrUw89T 2024-02-03T14:41:39.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:537: test_tiering_8: rm -f /tmp/tmp.9eMPrUw89T 2024-02-03T14:41:39.622 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:538: test_tiering_8: flush_pg_stats 2024-02-03T14:41:39.622 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-03T14:41:39.623 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-03T14:41:40.003 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-03T14:41:40.003 INFO:tasks.workunit.client.0.smithi007.stderr:1 2024-02-03T14:41:40.003 INFO:tasks.workunit.client.0.smithi007.stderr:2' 2024-02-03T14:41:40.003 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-03T14:41:40.003 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:41:40.004 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-03T14:41:40.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738418 2024-02-03T14:41:40.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738418 2024-02-03T14:41:40.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738418' 2024-02-03T14:41:40.158 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:41:40.158 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-03T14:41:40.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738418 2024-02-03T14:41:40.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738418 2024-02-03T14:41:40.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738418 1-34359738418' 2024-02-03T14:41:40.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-03T14:41:40.308 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-03T14:41:40.455 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=34359738418 2024-02-03T14:41:40.455 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 34359738418 2024-02-03T14:41:40.455 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-34359738418 1-34359738418 2-34359738418' 2024-02-03T14:41:40.456 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:41:40.456 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-34359738418 2024-02-03T14:41:40.456 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:41:40.456 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-03T14:41:40.457 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-34359738418 2024-02-03T14:41:40.457 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:41:40.458 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738418 2024-02-03T14:41:40.458 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 34359738418' 2024-02-03T14:41:40.458 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.0 seq 34359738418 2024-02-03T14:41:40.459 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:41:40.834 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738416 -lt 34359738418 2024-02-03T14:41:40.834 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-03T14:41:41.835 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-03T14:41:41.836 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-03T14:41:42.237 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738418 -lt 34359738418 2024-02-03T14:41:42.238 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:41:42.238 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:41:42.238 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-34359738418 2024-02-03T14:41:42.239 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-03T14:41:42.240 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-34359738418 2024-02-03T14:41:42.240 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:41:42.241 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738418 2024-02-03T14:41:42.241 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.1 seq 34359738418 2024-02-03T14:41:42.241 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 34359738418' 2024-02-03T14:41:42.241 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-03T14:41:42.619 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738418 -lt 34359738418 2024-02-03T14:41:42.619 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-03T14:41:42.620 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-34359738418 2024-02-03T14:41:42.620 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-03T14:41:42.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-03T14:41:42.621 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-34359738418 2024-02-03T14:41:42.622 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-03T14:41:42.623 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=34359738418 2024-02-03T14:41:42.623 INFO:tasks.workunit.client.0.smithi007.stdout:waiting osd.2 seq 34359738418 2024-02-03T14:41:42.624 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 34359738418' 2024-02-03T14:41:42.624 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-03T14:41:43.013 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2214: flush_pg_stats: test 34359738418 -lt 34359738418 2024-02-03T14:41:43.013 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: ceph df 2024-02-03T14:41:43.013 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: grep ' 2 ' 2024-02-03T14:41:43.013 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: grep datapool 2024-02-03T14:41:43.499 INFO:tasks.workunit.client.0.smithi007.stdout:datapool 19 2 0 B 2 0 B 0 128 GiB 2024-02-03T14:41:43.499 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:540: test_tiering_8: ceph osd tier remove-overlay datapool 2024-02-03T14:41:44.665 INFO:tasks.workunit.client.0.smithi007.stderr:there is now (or already was) no overlay for 'datapool' 2024-02-03T14:41:44.683 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:541: test_tiering_8: ceph osd tier remove datapool cache4 2024-02-03T14:41:45.672 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache4' is now (or already was) not a tier of 'datapool' 2024-02-03T14:41:45.689 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:542: test_tiering_8: ceph osd pool delete cache4 cache4 --yes-i-really-really-mean-it 2024-02-03T14:41:46.659 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache4' does not exist 2024-02-03T14:41:46.677 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:543: test_tiering_8: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2024-02-03T14:41:47.681 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'datapool' does not exist 2024-02-03T14:41:47.698 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:544: test_tiering_8: ceph osd unset notieragent 2024-02-03T14:41:49.599 INFO:tasks.workunit.client.0.smithi007.stderr:notieragent is unset 2024-02-03T14:41:49.620 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:41:50.019 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_9 2024-02-03T14:41:50.048 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:553: test_tiering_9: ceph osd pool create basepoolA 2 2024-02-03T14:41:50.705 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basepoolA' already exists 2024-02-03T14:41:50.722 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:554: test_tiering_9: ceph osd pool application enable basepoolA rados 2024-02-03T14:41:52.620 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'basepoolA' 2024-02-03T14:41:52.641 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:555: test_tiering_9: ceph osd pool create basepoolB 2 2024-02-03T14:41:53.729 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basepoolB' already exists 2024-02-03T14:41:53.747 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:556: test_tiering_9: ceph osd pool application enable basepoolB rados 2024-02-03T14:41:55.651 INFO:tasks.workunit.client.0.smithi007.stderr:enabled application 'rados' on pool 'basepoolB' 2024-02-03T14:41:55.676 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: ceph osd dump 2024-02-03T14:41:55.676 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: grep 'pool.*basepoolA' 2024-02-03T14:41:55.676 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: awk '{print $2;}' 2024-02-03T14:41:56.076 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: poolA_id=21 2024-02-03T14:41:56.077 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: grep 'pool.*basepoolB' 2024-02-03T14:41:56.077 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: ceph osd dump 2024-02-03T14:41:56.077 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: awk '{print $2;}' 2024-02-03T14:41:56.471 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: poolB_id=22 2024-02-03T14:41:56.471 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:560: test_tiering_9: ceph osd pool create cache5 2 2024-02-03T14:41:57.756 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache5' already exists 2024-02-03T14:41:57.773 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:561: test_tiering_9: ceph osd pool create cache6 2 2024-02-03T14:41:58.773 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache6' already exists 2024-02-03T14:41:58.789 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:562: test_tiering_9: ceph osd tier add basepoolA cache5 2024-02-03T14:41:59.795 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache5' is now (or already was) a tier of 'basepoolA' 2024-02-03T14:41:59.811 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:563: test_tiering_9: ceph osd tier add basepoolB cache6 2024-02-03T14:42:00.802 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache6' is now (or already was) a tier of 'basepoolB' 2024-02-03T14:42:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:564: test_tiering_9: ceph osd tier remove basepoolB cache5 2024-02-03T14:42:00.821 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:564: test_tiering_9: grep 'not a tier of' 2024-02-03T14:42:01.247 INFO:tasks.workunit.client.0.smithi007.stdout:pool 'cache5' is now (or already was) not a tier of 'basepoolB' 2024-02-03T14:42:01.247 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: ceph osd dump 2024-02-03T14:42:01.248 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: grep 'pool.*'\''cache5'\''' 2024-02-03T14:42:01.248 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: grep 'tier_of[ \t]\+21' 2024-02-03T14:42:01.640 INFO:tasks.workunit.client.0.smithi007.stdout:pool 23 'cache5' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 132 flags hashpspool tier_of 21 stripe_width 0 application rados read_balance_score 1.49 2024-02-03T14:42:01.641 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:566: test_tiering_9: ceph osd tier remove basepoolA cache6 2024-02-03T14:42:01.641 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:566: test_tiering_9: grep 'not a tier of' 2024-02-03T14:42:02.062 INFO:tasks.workunit.client.0.smithi007.stdout:pool 'cache6' is now (or already was) not a tier of 'basepoolA' 2024-02-03T14:42:02.062 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: ceph osd dump 2024-02-03T14:42:02.063 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: grep 'tier_of[ \t]\+22' 2024-02-03T14:42:02.063 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: grep 'pool.*'\''cache6'\''' 2024-02-03T14:42:02.449 INFO:tasks.workunit.client.0.smithi007.stdout:pool 24 'cache6' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 133 flags hashpspool tier_of 22 stripe_width 0 application rados read_balance_score 2.99 2024-02-03T14:42:02.450 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:569: test_tiering_9: ceph osd tier remove basepoolA cache5 2024-02-03T14:42:02.450 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:569: test_tiering_9: grep 'not a tier of' 2024-02-03T14:42:03.827 INFO:tasks.workunit.client.0.smithi007.stdout:pool 'cache5' is now (or already was) not a tier of 'basepoolA' 2024-02-03T14:42:03.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: ceph osd dump 2024-02-03T14:42:03.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: grep 'pool.*'\''cache5'\''' 2024-02-03T14:42:03.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: grep tier_of 2024-02-03T14:42:04.227 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:571: test_tiering_9: ceph osd tier remove basepoolB cache6 2024-02-03T14:42:04.227 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:571: test_tiering_9: grep 'not a tier of' 2024-02-03T14:42:05.181 INFO:tasks.workunit.client.0.smithi007.stdout:pool 'cache6' is now (or already was) not a tier of 'basepoolB' 2024-02-03T14:42:05.182 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: ceph osd dump 2024-02-03T14:42:05.182 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: grep 'pool.*'\''cache6'\''' 2024-02-03T14:42:05.182 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: grep tier_of 2024-02-03T14:42:05.573 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: grep 'pool.*'\''basepoolA'\''' 2024-02-03T14:42:05.573 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: ceph osd dump 2024-02-03T14:42:05.573 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: grep tiers 2024-02-03T14:42:05.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: ceph osd dump 2024-02-03T14:42:05.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: grep 'pool.*'\''basepoolB'\''' 2024-02-03T14:42:05.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: grep tiers 2024-02-03T14:42:06.368 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:577: test_tiering_9: ceph osd pool delete cache6 cache6 --yes-i-really-really-mean-it 2024-02-03T14:42:07.164 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache6' does not exist 2024-02-03T14:42:07.182 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:578: test_tiering_9: ceph osd pool delete cache5 cache5 --yes-i-really-really-mean-it 2024-02-03T14:42:08.182 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'cache5' does not exist 2024-02-03T14:42:08.200 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:579: test_tiering_9: ceph osd pool delete basepoolB basepoolB --yes-i-really-really-mean-it 2024-02-03T14:42:09.185 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basepoolB' does not exist 2024-02-03T14:42:09.203 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:580: test_tiering_9: ceph osd pool delete basepoolA basepoolA --yes-i-really-really-mean-it 2024-02-03T14:42:10.206 INFO:tasks.workunit.client.0.smithi007.stderr:pool 'basepoolA' does not exist 2024-02-03T14:42:10.223 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:42:10.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_auth 2024-02-03T14:42:10.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:585: test_auth: expect_false ceph auth add client.xx mon invalid osd 'allow *' 2024-02-03T14:42:10.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:42:10.621 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth add client.xx mon invalid osd 'allow *' 2024-02-03T14:42:10.973 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: mon capability parse failed, stopped at 'invalid' of 'invalid' 2024-02-03T14:42:10.978 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:42:10.978 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:586: test_auth: expect_false ceph auth add client.xx mon 'allow *' osd 'allow *' invalid 'allow *' 2024-02-03T14:42:10.978 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:42:10.978 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth add client.xx mon 'allow *' osd 'allow *' invalid 'allow *' 2024-02-03T14:42:11.330 INFO:tasks.workunit.client.0.smithi007.stderr:Error EINVAL: unknown cap type 'invalid' 2024-02-03T14:42:11.334 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:42:11.334 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:587: test_auth: ceph auth add client.xx mon 'allow *' osd 'allow *' 2024-02-03T14:42:11.844 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:588: test_auth: ceph auth export client.xx 2024-02-03T14:42:12.365 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:589: test_auth: ceph auth add client.xx -i client.xx.keyring 2024-02-03T14:42:12.858 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:590: test_auth: rm -f client.xx.keyring 2024-02-03T14:42:12.860 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:591: test_auth: ceph auth list 2024-02-03T14:42:12.860 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:591: test_auth: grep client.xx 2024-02-03T14:42:13.355 INFO:tasks.workunit.client.0.smithi007.stdout:client.xx 2024-02-03T14:42:13.356 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:592: test_auth: ceph auth ls 2024-02-03T14:42:13.356 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:592: test_auth: grep client.xx 2024-02-03T14:42:13.844 INFO:tasks.workunit.client.0.smithi007.stdout:client.xx 2024-02-03T14:42:13.845 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: ceph auth get client.xx 2024-02-03T14:42:13.845 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: grep caps 2024-02-03T14:42:13.845 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: grep mon 2024-02-03T14:42:14.341 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow *" 2024-02-03T14:42:14.342 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: ceph auth get client.xx 2024-02-03T14:42:14.342 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: grep caps 2024-02-03T14:42:14.342 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: grep osd 2024-02-03T14:42:14.822 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:14.822 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:595: test_auth: ceph auth get-key client.xx 2024-02-03T14:42:15.323 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:596: test_auth: ceph auth print-key client.xx 2024-02-03T14:42:15.819 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:597: test_auth: ceph auth print_key client.xx 2024-02-03T14:42:16.318 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:598: test_auth: ceph auth caps client.xx osd 'allow rw' 2024-02-03T14:42:16.812 INFO:tasks.workunit.client.0.smithi007.stdout:AQDDUL5lUgp9KBAA8rNsZul/IiMSdu22lH145w==AQDDUL5lUgp9KBAA8rNsZul/IiMSdu22lH145w==AQDDUL5lUgp9KBAA8rNsZul/IiMSdu22lH145w==[client.xx] 2024-02-03T14:42:16.812 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDDUL5lUgp9KBAA8rNsZul/IiMSdu22lH145w== 2024-02-03T14:42:16.813 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow rw" 2024-02-03T14:42:16.813 INFO:tasks.workunit.client.0.smithi007.stderr:updated caps for client.xx 2024-02-03T14:42:16.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:599: test_auth: expect_false sh 2024-02-03T14:42:16.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:42:16.828 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: sh 2024-02-03T14:42:17.331 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:42:17.332 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: grep osd 2024-02-03T14:42:17.332 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: ceph auth get client.xx 2024-02-03T14:42:17.332 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: grep 'allow rw' 2024-02-03T14:42:17.822 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow rw" 2024-02-03T14:42:17.822 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:601: test_auth: ceph auth caps client.xx mon 'allow command "osd tree"' 2024-02-03T14:42:18.317 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx] 2024-02-03T14:42:18.317 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDDUL5lUgp9KBAA8rNsZul/IiMSdu22lH145w== 2024-02-03T14:42:18.317 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow command \"osd tree\"" 2024-02-03T14:42:18.317 INFO:tasks.workunit.client.0.smithi007.stderr:updated caps for client.xx 2024-02-03T14:42:18.333 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:602: test_auth: ceph auth export 2024-02-03T14:42:18.333 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:602: test_auth: grep client.xx 2024-02-03T14:42:18.830 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx] 2024-02-03T14:42:18.831 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:603: test_auth: ceph auth export -o authfile 2024-02-03T14:42:19.327 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:604: test_auth: ceph auth import -i authfile 2024-02-03T14:42:19.824 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:606: test_auth: ceph auth export -o authfile2 2024-02-03T14:42:20.299 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:607: test_auth: diff authfile authfile2 2024-02-03T14:42:20.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:608: test_auth: rm authfile authfile2 2024-02-03T14:42:20.308 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:609: test_auth: ceph auth del client.xx 2024-02-03T14:42:20.818 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:610: test_auth: expect_false ceph auth get client.xx 2024-02-03T14:42:20.818 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:42:20.819 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth get client.xx 2024-02-03T14:42:21.152 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOENT: failed to find client.xx in keyring 2024-02-03T14:42:21.156 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:42:21.157 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth: echo -e 'auth add client.xx mon "allow *" osd "allow *"\n' 2024-02-03T14:42:21.157 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth: ceph 2024-02-03T14:42:21.482 INFO:tasks.workunit.client.0.smithi007.stderr:added key for client.xx 2024-02-03T14:42:21.497 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:614: test_auth: ceph auth get client.xx 2024-02-03T14:42:21.965 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx] 2024-02-03T14:42:21.965 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDNUL5lxKWwHBAAHrFdupZUx7VP6OurLOs1Ww== 2024-02-03T14:42:21.965 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow *" 2024-02-03T14:42:21.965 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:21.980 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:616: test_auth: echo 'auth del client.xx' 2024-02-03T14:42:21.981 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:616: test_auth: ceph 2024-02-03T14:42:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:617: test_auth: expect_false ceph auth get client.xx 2024-02-03T14:42:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2024-02-03T14:42:22.330 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth get client.xx 2024-02-03T14:42:22.668 INFO:tasks.workunit.client.0.smithi007.stderr:Error ENOENT: failed to find client.xx in keyring 2024-02-03T14:42:22.673 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2024-02-03T14:42:22.673 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:42:23.071 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_auth_profiles 2024-02-03T14:42:23.071 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:622: test_auth_profiles: ceph auth add client.xx-profile-ro mon 'allow profile read-only' mgr 'allow profile read-only' 2024-02-03T14:42:23.587 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:624: test_auth_profiles: ceph auth add client.xx-profile-rw mon 'allow profile read-write' mgr 'allow profile read-write' 2024-02-03T14:42:24.100 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:626: test_auth_profiles: ceph auth add client.xx-profile-rd mon 'allow profile role-definer' 2024-02-03T14:42:24.595 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:628: test_auth_profiles: ceph auth export 2024-02-03T14:42:25.080 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:631: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring status 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: cluster: 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: id: 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: health: HEALTH_OK 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: services: 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2024-02-03T14:42:25.567 INFO:tasks.workunit.client.0.smithi007.stdout: mgr: x(active, since 4m) 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: data: 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: pools: 2 pools, 9 pgs 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: objects: 2 objects, 577 KiB 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: usage: 99 MiB used, 270 GiB / 270 GiB avail 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: pgs: 9 active+clean 2024-02-03T14:42:25.568 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:25.583 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:632: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring osd dump 2024-02-03T14:42:25.956 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 139 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:created 2024-02-03T14:37:58.278765+0000 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:modified 2024-02-03T14:42:10.091436+0000 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:crush_version 5 2024-02-03T14:42:25.957 INFO:tasks.workunit.client.0.smithi007.stdout:full_ratio 0.95 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:backfillfull_ratio 0.9 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:nearfull_ratio 0.85 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:require_min_compat_client luminous 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:min_compat_client jewel 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:require_osd_release squid 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:stretch_mode_enabled false 2024-02-03T14:42:25.958 INFO:tasks.workunit.client.0.smithi007.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 application rbd read_balance_score 1.88 2024-02-03T14:42:25.959 INFO:tasks.workunit.client.0.smithi007.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.99 2024-02-03T14:42:25.959 INFO:tasks.workunit.client.0.smithi007.stdout:max_osd 3 2024-02-03T14:42:25.959 INFO:tasks.workunit.client.0.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6808/1938254861 v1:172.21.15.7:6809/1938254861 exists,up 6eff2159-02fa-462e-937d-98e2359ecd93 2024-02-03T14:42:25.959 INFO:tasks.workunit.client.0.smithi007.stdout:osd.1 up in weight 1 up_from 8 up_thru 131 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6800/2463192558 v1:172.21.15.7:6801/2463192558 exists,up 428cfc4f-5eb3-4048-b602-43dd7d453c2d 2024-02-03T14:42:25.959 INFO:tasks.workunit.client.0.smithi007.stdout:osd.2 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6804/4029165148 v1:172.21.15.7:6805/4029165148 exists,up 563c4d0d-9b1e-470d-aaa3-57a4ca92b336 2024-02-03T14:42:25.973 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:633: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring pg dump 2024-02-03T14:42:26.342 INFO:tasks.workunit.client.0.smithi007.stdout:version 267 2024-02-03T14:42:26.342 INFO:tasks.workunit.client.0.smithi007.stdout:stamp 2024-02-03T14:42:25.224114+0000 2024-02-03T14:42:26.342 INFO:tasks.workunit.client.0.smithi007.stdout:last_osdmap_epoch 0 2024-02-03T14:42:26.342 INFO:tasks.workunit.client.0.smithi007.stdout:last_pg_scan 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.279819+0000 0'0 139:141 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.280826+0000 0'0 139:133 [2,0] 2 [2,0] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.284793+0000 0'0 139:126 [0,1] 0 [0,1] 0 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.281552+0000 0'0 139:141 [1,2] 1 [1,2] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:2.0 2 0 0 0 0 590368 0 0 97 0 97 active+clean 2024-02-03T14:38:08.281706+0000 10'97 139:260 [2,1] 2 [2,1] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.284531+0000 0'0 139:141 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.283182+0000 0'0 139:133 [2,0] 2 [2,0] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.282915+0000 0'0 138:140 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.281448+0000 0'0 138:140 [1,2] 1 [1,2] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:2 2 0 0 0 0 590368 0 0 97 97 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout:1 0 0 0 0 0 0 0 0 0 0 2024-02-03T14:42:26.343 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:sum 2 0 0 0 0 590368 0 0 97 97 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:2 28 MiB 90 GiB 28 MiB 90 GiB [0,1] 5 3 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:1 36 MiB 90 GiB 36 MiB 90 GiB [0,2] 7 5 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:0 36 MiB 90 GiB 36 MiB 90 GiB [1,2] 6 1 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:sum 99 MiB 270 GiB 99 MiB 270 GiB 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-03T14:42:26.344 INFO:tasks.workunit.client.0.smithi007.stderr:dumped all 2024-02-03T14:42:26.357 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:634: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring mon dump 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 1 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:last_changed 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:created 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:min_mon_release 19 (squid) 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:election_strategy: 1 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:0: v1:172.21.15.7:6789/0 mon.a 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:1: v1:172.21.15.7:6790/0 mon.b 2024-02-03T14:42:26.849 INFO:tasks.workunit.client.0.smithi007.stdout:2: v1:172.21.15.7:6791/0 mon.c 2024-02-03T14:42:26.850 INFO:tasks.workunit.client.0.smithi007.stderr:dumped monmap epoch 1 2024-02-03T14:42:26.864 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:636: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring log foo 2024-02-03T14:42:27.212 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:636: test_auth_profiles: true 2024-02-03T14:42:27.212 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:637: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:27.213 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:638: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring osd set noout 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:638: test_auth_profiles: true 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:639: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:27.508 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:27.509 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:640: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring auth ls 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:640: test_auth_profiles: true 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:641: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:27.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:27.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:27.862 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:644: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring status 2024-02-03T14:42:28.351 INFO:tasks.workunit.client.0.smithi007.stdout: cluster: 2024-02-03T14:42:28.351 INFO:tasks.workunit.client.0.smithi007.stdout: id: 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:28.351 INFO:tasks.workunit.client.0.smithi007.stdout: health: HEALTH_OK 2024-02-03T14:42:28.351 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: services: 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: mgr: x(active, since 4m) 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: data: 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: pools: 2 pools, 9 pgs 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: objects: 2 objects, 577 KiB 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: usage: 99 MiB used, 270 GiB / 270 GiB avail 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: pgs: 9 active+clean 2024-02-03T14:42:28.352 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:28.367 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:645: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd dump 2024-02-03T14:42:28.747 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 139 2024-02-03T14:42:28.747 INFO:tasks.workunit.client.0.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:created 2024-02-03T14:37:58.278765+0000 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:modified 2024-02-03T14:42:10.091436+0000 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:crush_version 5 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:full_ratio 0.95 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:backfillfull_ratio 0.9 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:nearfull_ratio 0.85 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:require_min_compat_client luminous 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:min_compat_client jewel 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:require_osd_release squid 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:stretch_mode_enabled false 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 application rbd read_balance_score 1.88 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.99 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:max_osd 3 2024-02-03T14:42:28.748 INFO:tasks.workunit.client.0.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6808/1938254861 v1:172.21.15.7:6809/1938254861 exists,up 6eff2159-02fa-462e-937d-98e2359ecd93 2024-02-03T14:42:28.749 INFO:tasks.workunit.client.0.smithi007.stdout:osd.1 up in weight 1 up_from 8 up_thru 131 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6800/2463192558 v1:172.21.15.7:6801/2463192558 exists,up 428cfc4f-5eb3-4048-b602-43dd7d453c2d 2024-02-03T14:42:28.749 INFO:tasks.workunit.client.0.smithi007.stdout:osd.2 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) v1:172.21.15.7:6804/4029165148 v1:172.21.15.7:6805/4029165148 exists,up 563c4d0d-9b1e-470d-aaa3-57a4ca92b336 2024-02-03T14:42:28.766 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:646: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring pg dump 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:version 268 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:stamp 2024-02-03T14:42:27.224373+0000 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:last_osdmap_epoch 0 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:last_pg_scan 0 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-03T14:42:29.140 INFO:tasks.workunit.client.0.smithi007.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.279819+0000 0'0 139:141 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.280826+0000 0'0 139:133 [2,0] 2 [2,0] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.284793+0000 0'0 139:126 [0,1] 0 [0,1] 0 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.281552+0000 0'0 139:141 [1,2] 1 [1,2] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:2.0 2 0 0 0 0 590368 0 0 97 0 97 active+clean 2024-02-03T14:38:08.281706+0000 10'97 139:260 [2,1] 2 [2,1] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.284531+0000 0'0 139:141 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.283182+0000 0'0 139:133 [2,0] 2 [2,0] 2 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.282915+0000 0'0 139:141 [1,0] 1 [1,0] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-03T14:38:08.281448+0000 0'0 138:140 [1,2] 1 [1,2] 1 0'0 2024-02-03T14:38:07.262531+0000 0'0 2024-02-03T14:38:07.262531+0000 0 0 periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:2 2 0 0 0 0 590368 0 0 97 97 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout:1 0 0 0 0 0 0 0 0 0 0 2024-02-03T14:42:29.141 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:sum 2 0 0 0 0 590368 0 0 97 97 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:2 28 MiB 90 GiB 28 MiB 90 GiB [0,1] 5 3 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:1 36 MiB 90 GiB 36 MiB 90 GiB [0,2] 7 5 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:0 36 MiB 90 GiB 36 MiB 90 GiB [1,2] 6 1 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:sum 99 MiB 270 GiB 99 MiB 270 GiB 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-03T14:42:29.142 INFO:tasks.workunit.client.0.smithi007.stderr:dumped all 2024-02-03T14:42:29.155 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:647: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring mon dump 2024-02-03T14:42:29.640 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 1 2024-02-03T14:42:29.641 INFO:tasks.workunit.client.0.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:29.641 INFO:tasks.workunit.client.0.smithi007.stdout:last_changed 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:29.641 INFO:tasks.workunit.client.0.smithi007.stdout:created 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:29.641 INFO:tasks.workunit.client.0.smithi007.stdout:min_mon_release 19 (squid) 2024-02-03T14:42:29.641 INFO:tasks.workunit.client.0.smithi007.stdout:election_strategy: 1 2024-02-03T14:42:29.642 INFO:tasks.workunit.client.0.smithi007.stdout:0: v1:172.21.15.7:6789/0 mon.a 2024-02-03T14:42:29.642 INFO:tasks.workunit.client.0.smithi007.stdout:1: v1:172.21.15.7:6790/0 mon.b 2024-02-03T14:42:29.642 INFO:tasks.workunit.client.0.smithi007.stdout:2: v1:172.21.15.7:6791/0 mon.c 2024-02-03T14:42:29.642 INFO:tasks.workunit.client.0.smithi007.stderr:dumped monmap epoch 1 2024-02-03T14:42:29.657 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:648: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring fs dump 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout:e1 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout:legacy client fscid: -1 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:30.099 INFO:tasks.workunit.client.0.smithi007.stdout:No filesystems configured 2024-02-03T14:42:30.100 INFO:tasks.workunit.client.0.smithi007.stderr:dumped fsmap epoch 1 2024-02-03T14:42:30.115 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:649: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring log foo 2024-02-03T14:42:32.211 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:650: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd set noout 2024-02-03T14:42:34.212 INFO:tasks.workunit.client.0.smithi007.stderr:noout is set 2024-02-03T14:42:34.231 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:651: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd unset noout 2024-02-03T14:42:36.227 INFO:tasks.workunit.client.0.smithi007.stderr:noout is unset 2024-02-03T14:42:36.247 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:653: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring auth ls 2024-02-03T14:42:36.593 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:653: test_auth_profiles: true 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:654: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:36.594 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:657: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth ls 2024-02-03T14:42:37.080 INFO:tasks.workunit.client.0.smithi007.stdout:osd.0 2024-02-03T14:42:37.080 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC2T75l4LjXFxAAUJlAtWONul7R3XuQuJfLLw== 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow profile osd 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile osd 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow * 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout:osd.1 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC5T75lanuMHBAActgm8MocUpn/yvD4WPNyYg== 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow profile osd 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile osd 2024-02-03T14:42:37.081 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow * 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout:osd.2 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC8T75l2uKWIxAA/bHZMZz0KSGxoa5YDmE7dA== 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow profile osd 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile osd 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow * 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout:client.0 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC1T75l9dgzHxAASJCXGkpTzrd2+8T1SWvvng== 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mds] allow 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow r 2024-02-03T14:42:37.082 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow rw 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow rwx 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout:client.admin 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC1T75lqHluCxAAhkvdP6FmCF/8UnMhdesKKw== 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mds] allow * 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow * 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow * 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow * 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-mds 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lLQa0EBAA3WC3ZgfCdnBupLchznbzSg== 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-mds 2024-02-03T14:42:37.083 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-mgr 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lzzG0EBAA5pY2KQ108JthXVhRJO1EGA== 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-mgr 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-osd 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lDVO0EBAACuHXQcziKtoxj3nFtAY6nA== 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-osd 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-rbd 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lXGu0EBAAE6jlEe89H/wCRAQMCWsXdw== 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-rbd 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-rbd-mirror 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lz4K0EBAA12oZobeQIqIuoXulGOWbZQ== 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-rbd-mirror 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout:client.bootstrap-rgw 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDGT75lfpm0EBAAe3kU563hwW+V9IT62l+/+A== 2024-02-03T14:42:37.084 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile bootstrap-rgw 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout:client.xx-profile-rd 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDQUL5lAvnHGRAA1rX5THYjpqhqpC1qngwm1g== 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile role-definer 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout:client.xx-profile-ro 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDPUL5l6OLwGBAA8dij2p61VRcrrLTzIcAlww== 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow profile read-only 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile read-only 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout:client.xx-profile-rw 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQDPUL5lXa8BNxAAbEyjIxnKg3q46Phby9n+UQ== 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mgr] allow profile read-write 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile read-write 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout:mgr.x 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: key: AQC1T75lINp/GhAAivNmMlmkLHdbTbi0xZSfag== 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mds] allow * 2024-02-03T14:42:37.085 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [mon] allow profile mgr 2024-02-03T14:42:37.086 INFO:tasks.workunit.client.0.smithi007.stdout: caps: [osd] allow * 2024-02-03T14:42:37.096 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:658: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth export 2024-02-03T14:42:37.559 INFO:tasks.workunit.client.0.smithi007.stdout:[osd.0] 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC2T75l4LjXFxAAUJlAtWONul7R3XuQuJfLLw== 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow profile osd" 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile osd" 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout:[osd.1] 2024-02-03T14:42:37.560 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC5T75lanuMHBAActgm8MocUpn/yvD4WPNyYg== 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow profile osd" 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile osd" 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout:[osd.2] 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC8T75l2uKWIxAA/bHZMZz0KSGxoa5YDmE7dA== 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow profile osd" 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile osd" 2024-02-03T14:42:37.561 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout:[client.0] 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC1T75l9dgzHxAASJCXGkpTzrd2+8T1SWvvng== 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mds = "allow" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow r" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow rw" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow rwx" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout:[client.admin] 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC1T75lqHluCxAAhkvdP6FmCF/8UnMhdesKKw== 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mds = "allow *" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow *" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow *" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:37.562 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-mds] 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lLQa0EBAA3WC3ZgfCdnBupLchznbzSg== 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-mds" 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-mgr] 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lzzG0EBAA5pY2KQ108JthXVhRJO1EGA== 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-mgr" 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-osd] 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lDVO0EBAACuHXQcziKtoxj3nFtAY6nA== 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-osd" 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-rbd] 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lXGu0EBAAE6jlEe89H/wCRAQMCWsXdw== 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-rbd" 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-rbd-mirror] 2024-02-03T14:42:37.563 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lz4K0EBAA12oZobeQIqIuoXulGOWbZQ== 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-rbd-mirror" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout:[client.bootstrap-rgw] 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDGT75lfpm0EBAAe3kU563hwW+V9IT62l+/+A== 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile bootstrap-rgw" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx-profile-rd] 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDQUL5lAvnHGRAA1rX5THYjpqhqpC1qngwm1g== 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile role-definer" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx-profile-ro] 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDPUL5l6OLwGBAA8dij2p61VRcrrLTzIcAlww== 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow profile read-only" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile read-only" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout:[client.xx-profile-rw] 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQDPUL5lXa8BNxAAbEyjIxnKg3q46Phby9n+UQ== 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mgr = "allow profile read-write" 2024-02-03T14:42:37.564 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile read-write" 2024-02-03T14:42:37.565 INFO:tasks.workunit.client.0.smithi007.stdout:[mgr.x] 2024-02-03T14:42:37.565 INFO:tasks.workunit.client.0.smithi007.stdout: key = AQC1T75lINp/GhAAivNmMlmkLHdbTbi0xZSfag== 2024-02-03T14:42:37.565 INFO:tasks.workunit.client.0.smithi007.stdout: caps mds = "allow *" 2024-02-03T14:42:37.565 INFO:tasks.workunit.client.0.smithi007.stdout: caps mon = "allow profile mgr" 2024-02-03T14:42:37.565 INFO:tasks.workunit.client.0.smithi007.stdout: caps osd = "allow *" 2024-02-03T14:42:37.572 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:659: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth add client.xx-profile-foo 2024-02-03T14:42:38.083 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:660: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring status 2024-02-03T14:42:38.546 INFO:tasks.workunit.client.0.smithi007.stdout: cluster: 2024-02-03T14:42:38.546 INFO:tasks.workunit.client.0.smithi007.stdout: id: 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:38.546 INFO:tasks.workunit.client.0.smithi007.stdout: health: HEALTH_OK 2024-02-03T14:42:38.546 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: services: 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: mgr: x(active, since 4m) 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: data: 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: pools: 2 pools, 9 pgs 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: objects: 2 objects, 577 KiB 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: usage: 99 MiB used, 270 GiB / 270 GiB avail 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: pgs: 9 active+clean 2024-02-03T14:42:38.547 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:38.562 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:661: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring osd dump 2024-02-03T14:42:38.842 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:661: test_auth_profiles: true 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:662: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:38.843 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:38.844 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:663: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring pg dump 2024-02-03T14:42:39.116 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-02-03T14:42:39.116+0000 7fd207411640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2024-02-03T14:42:39.121 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:663: test_auth_profiles: true 2024-02-03T14:42:39.121 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:664: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:39.121 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:39.121 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:39.122 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:39.122 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:39.122 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:39.122 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:666: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring mon dump 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 1 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:fsid 2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:last_changed 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:created 2024-02-03T14:37:40.964483+0000 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:min_mon_release 19 (squid) 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:election_strategy: 1 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:0: v1:172.21.15.7:6789/0 mon.a 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:1: v1:172.21.15.7:6790/0 mon.b 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stdout:2: v1:172.21.15.7:6791/0 mon.c 2024-02-03T14:42:39.604 INFO:tasks.workunit.client.0.smithi007.stderr:dumped monmap epoch 1 2024-02-03T14:42:39.619 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:668: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring mon add foo 1.1.1.1 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:668: test_auth_profiles: true 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:669: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:39.970 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:39.971 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:670: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring fs dump 2024-02-03T14:42:40.295 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:670: test_auth_profiles: true 2024-02-03T14:42:40.295 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:671: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:40.296 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:672: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring log foo 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:672: test_auth_profiles: true 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:673: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:40.635 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:40.636 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:674: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring osd set noout 2024-02-03T14:42:40.931 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:674: test_auth_profiles: true 2024-02-03T14:42:40.931 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:675: test_auth_profiles: check_response 'EACCES: access denied' 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:40.932 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:677: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth del client.xx-profile-ro 2024-02-03T14:42:41.449 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:678: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth del client.xx-profile-rw 2024-02-03T14:42:41.973 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:681: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth add client.xx-profile-rd2 mon 'allow profile role-definer' 2024-02-03T14:42:42.469 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:683: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth export 2024-02-03T14:42:42.969 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:686: test_auth_profiles: ceph -n client.xx-profile-rd2 -k client.xx.keyring.2 auth del client.xx-profile-rd 2024-02-03T14:42:43.480 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:689: test_auth_profiles: ceph auth del client.xx-profile-rd2 2024-02-03T14:42:43.994 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:690: test_auth_profiles: rm -f client.xx.keyring client.xx.keyring.2 2024-02-03T14:42:43.995 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2024-02-03T14:42:44.380 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_misc 2024-02-03T14:42:44.380 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:719: test_mon_misc: ceph osd dump 2024-02-03T14:42:44.380 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:719: test_mon_misc: grep '^epoch' 2024-02-03T14:42:44.772 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 143 2024-02-03T14:42:44.772 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:720: test_mon_misc: ceph --concise osd dump 2024-02-03T14:42:44.772 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:720: test_mon_misc: grep '^epoch' 2024-02-03T14:42:45.151 INFO:tasks.workunit.client.0.smithi007.stdout:epoch 143 2024-02-03T14:42:45.151 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:722: test_mon_misc: ceph osd df 2024-02-03T14:42:45.151 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:722: test_mon_misc: grep 'MIN/MAX VAR' 2024-02-03T14:42:45.532 INFO:tasks.workunit.client.0.smithi007.stdout:MIN/MAX VAR: 0.84/1.08 STDDEV: 0.00 2024-02-03T14:42:45.532 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:725: test_mon_misc: ceph df 2024-02-03T14:42:46.016 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:726: test_mon_misc: grep RAW /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:46.018 INFO:tasks.workunit.client.0.smithi007.stdout:--- RAW STORAGE --- 2024-02-03T14:42:46.018 INFO:tasks.workunit.client.0.smithi007.stdout:CLASS SIZE AVAIL USED RAW USED %RAW USED 2024-02-03T14:42:46.018 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:727: test_mon_misc: grep -v DIRTY /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:46.019 INFO:tasks.workunit.client.0.smithi007.stdout:--- RAW STORAGE --- 2024-02-03T14:42:46.019 INFO:tasks.workunit.client.0.smithi007.stdout:CLASS SIZE AVAIL USED RAW USED %RAW USED 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:ssd 270 GiB 270 GiB 99 MiB 99 MiB 0.04 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:TOTAL 270 GiB 270 GiB 99 MiB 99 MiB 0.04 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:--- POOLS --- 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:rbd 1 8 0 B 0 0 B 0 128 GiB 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:.mgr 2 1 456 KiB 2 104 KiB 0 128 GiB 2024-02-03T14:42:46.020 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:728: test_mon_misc: ceph df detail 2024-02-03T14:42:46.523 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:729: test_mon_misc: grep DIRTY /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:46.524 INFO:tasks.workunit.client.0.smithi007.stdout:POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR 2024-02-03T14:42:46.524 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:730: test_mon_misc: ceph df --format json 2024-02-03T14:42:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:731: test_mon_misc: grep total_bytes /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:47.023 INFO:tasks.workunit.client.0.smithi007.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895}},"pools":[{"name":"rbd","id":1,"stats":{"stored":0,"objects":0,"kb_used":0,"bytes_used":0,"percent_used":0,"max_avail":137651109888}},{"name":".mgr","id":2,"stats":{"stored":466956,"objects":2,"kb_used":104,"bytes_used":106496,"percent_used":3.8683290881635912e-07,"max_avail":137651109888}}]} 2024-02-03T14:42:47.024 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:732: test_mon_misc: grep -v dirty /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:47.025 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:47.025 INFO:tasks.workunit.client.0.smithi007.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895}},"pools":[{"name":"rbd","id":1,"stats":{"stored":0,"objects":0,"kb_used":0,"bytes_used":0,"percent_used":0,"max_avail":137651109888}},{"name":".mgr","id":2,"stats":{"stored":466956,"objects":2,"kb_used":104,"bytes_used":106496,"percent_used":3.8683290881635912e-07,"max_avail":137651109888}}]} 2024-02-03T14:42:47.025 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:733: test_mon_misc: ceph df detail --format json 2024-02-03T14:42:47.510 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:734: test_mon_misc: grep rd_bytes /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:47.512 INFO:tasks.workunit.client.0.smithi007.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895}},"pools":[{"name":"rbd","id":1,"stats":{"stored":0,"stored_data":0,"stored_omap":0,"objects":0,"kb_used":0,"bytes_used":0,"data_bytes_used":0,"omap_bytes_used":0,"percent_used":0,"max_avail":137651109888,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":0,"rd_bytes":0,"wr":0,"wr_bytes":0,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":0,"avail_raw":275302220403}},{"name":".mgr","id":2,"stats":{"stored":466956,"stored_data":466956,"stored_omap":0,"objects":2,"kb_used":104,"bytes_used":106496,"data_bytes_used":106496,"omap_bytes_used":0,"percent_used":3.8683290881635912e-07,"max_avail":137651109888,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":106,"rd_bytes":93184,"wr":137,"wr_bytes":1939456,"compress_bytes_used":65536,"compress_under_bytes":901120,"stored_raw":933912,"avail_raw":275302220403}}]} 2024-02-03T14:42:47.512 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:735: test_mon_misc: grep dirty /tmp/cephtool.p0s/test_invalid.I40 2024-02-03T14:42:47.513 INFO:tasks.workunit.client.0.smithi007.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289806323712,"total_used_bytes":103968768,"total_used_raw_bytes":103968768,"total_used_raw_ratio":0.000358623918145895}},"pools":[{"name":"rbd","id":1,"stats":{"stored":0,"stored_data":0,"stored_omap":0,"objects":0,"kb_used":0,"bytes_used":0,"data_bytes_used":0,"omap_bytes_used":0,"percent_used":0,"max_avail":137651109888,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":0,"rd_bytes":0,"wr":0,"wr_bytes":0,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":0,"avail_raw":275302220403}},{"name":".mgr","id":2,"stats":{"stored":466956,"stored_data":466956,"stored_omap":0,"objects":2,"kb_used":104,"bytes_used":106496,"data_bytes_used":106496,"omap_bytes_used":0,"percent_used":3.8683290881635912e-07,"max_avail":137651109888,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":106,"rd_bytes":93184,"wr":137,"wr_bytes":1939456,"compress_bytes_used":65536,"compress_under_bytes":901120,"stored_raw":933912,"avail_raw":275302220403}}]} 2024-02-03T14:42:47.514 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:736: test_mon_misc: ceph df --format xml 2024-02-03T14:42:47.514 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:736: test_mon_misc: grep '' 2024-02-03T14:42:47.995 INFO:tasks.workunit.client.0.smithi007.stdout:2899102924802898063237121039687681039687680.0003586239181458953332899102924802898063237121039687681039687680.000358623918145895rbd100000137651109888.mgr246695621041064963.8683290881635912e-07137651109888 2024-02-03T14:42:47.996 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:737: test_mon_misc: ceph df detail --format xml 2024-02-03T14:42:47.996 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:737: test_mon_misc: grep '' 2024-02-03T14:42:48.489 INFO:tasks.workunit.client.0.smithi007.stdout:2899102924802898063237121039687681039687680.0003586239181458953332899102924802898063237121039687681039687680.000358623918145895rbd10000000001376511098880000000000275302220403.mgr24669564669560210410649610649603.8683290881635912e-0713765110988800010693184137193945665536901120933912275302220403 2024-02-03T14:42:48.489 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:739: test_mon_misc: ceph fsid 2024-02-03T14:42:48.972 INFO:tasks.workunit.client.0.smithi007.stdout:2dbfabd7-19a5-49d9-9435-d2e80516e779 2024-02-03T14:42:48.988 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:740: test_mon_misc: ceph health 2024-02-03T14:42:49.428 INFO:tasks.workunit.client.0.smithi007.stdout:HEALTH_OK 2024-02-03T14:42:49.444 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:741: test_mon_misc: ceph health detail 2024-02-03T14:42:49.922 INFO:tasks.workunit.client.0.smithi007.stdout:HEALTH_OK 2024-02-03T14:42:49.938 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:742: test_mon_misc: ceph health --format json-pretty 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout: "status": "HEALTH_OK", 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout: "checks": {}, 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout: "mutes": [] 2024-02-03T14:42:50.378 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:50.393 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:743: test_mon_misc: ceph health detail --format xml-pretty 2024-02-03T14:42:50.870 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: HEALTH_OK 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.871 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-02-03T14:42:50.885 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:745: test_mon_misc: ceph time-sync-status 2024-02-03T14:42:51.364 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:51.394 INFO:tasks.workunit.client.0.smithi007.stdout: "time_skew_status": { 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "a": { 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "skew": 0, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "latency": 0, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "health": "HEALTH_OK" 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: }, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "b": { 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "skew": 0, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "latency": 0.0011778540000000001, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "health": "HEALTH_OK" 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: }, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "c": { 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "skew": 0, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "latency": 0.001506131, 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: "health": "HEALTH_OK" 2024-02-03T14:42:51.395 INFO:tasks.workunit.client.0.smithi007.stdout: } 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: }, 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: "timechecks": { 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: "epoch": 8, 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: "round": 2, 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: "round_status": "finished" 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout: } 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:51.396 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:747: test_mon_misc: ceph node ls 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "mon": { 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "a", 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "b", 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "c" 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: }, 2024-02-03T14:42:51.842 INFO:tasks.workunit.client.0.smithi007.stdout: "osd": { 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: 0, 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: 1, 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: 2 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: }, 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: "mgr": { 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: "x" 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout: } 2024-02-03T14:42:51.843 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:51.857 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2024-02-03T14:42:51.857 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mon 2024-02-03T14:42:52.344 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:52.344 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:52.344 INFO:tasks.workunit.client.0.smithi007.stdout: "a", 2024-02-03T14:42:52.344 INFO:tasks.workunit.client.0.smithi007.stdout: "b", 2024-02-03T14:42:52.345 INFO:tasks.workunit.client.0.smithi007.stdout: "c" 2024-02-03T14:42:52.345 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:52.345 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:52.361 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2024-02-03T14:42:52.361 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls osd 2024-02-03T14:42:52.855 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout: 0, 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout: 1, 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout: 2 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:52.856 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:52.871 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2024-02-03T14:42:52.871 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mds 2024-02-03T14:42:53.370 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2024-02-03T14:42:53.370 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mgr 2024-02-03T14:42:53.835 INFO:tasks.workunit.client.0.smithi007.stdout:{ 2024-02-03T14:42:53.835 INFO:tasks.workunit.client.0.smithi007.stdout: "smithi007": [ 2024-02-03T14:42:53.835 INFO:tasks.workunit.client.0.smithi007.stdout: "x" 2024-02-03T14:42:53.835 INFO:tasks.workunit.client.0.smithi007.stdout: ] 2024-02-03T14:42:53.835 INFO:tasks.workunit.client.0.smithi007.stdout:} 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:752: test_mon_misc: ceph_watch_start 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:155: ceph_watch_start: local whatch_opt=--watch 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:157: ceph_watch_start: '[' -n '' ']' 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:164: ceph_watch_start: CEPH_WATCH_FILE=/tmp/cephtool.p0s/CEPH_WATCH_27256 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:166: ceph_watch_start: CEPH_WATCH_PID=35050 2024-02-03T14:42:53.851 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:165: ceph_watch_start: ceph --watch 2024-02-03T14:42:53.852 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: seq 3 2024-02-03T14:42:53.853 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2024-02-03T14:42:53.853 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.p0s/CEPH_WATCH_27256 2024-02-03T14:42:53.854 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2024-02-03T14:42:54.856 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2024-02-03T14:42:54.856 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.p0s/CEPH_WATCH_27256 2024-02-03T14:42:54.858 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2024-02-03T14:42:55.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2024-02-03T14:42:55.861 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.p0s/CEPH_WATCH_27256 2024-02-03T14:42:55.863 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2024-02-03T14:42:56.866 INFO:tasks.workunit.client.0.smithi007.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:753: test_mon_misc: date 2024-02-03T14:42:56.868 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:753: test_mon_misc: mymsg='this is a test log message 27256.Sat Feb 3 02:42:56 PM UTC 2024' 2024-02-03T14:42:56.868 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:754: test_mon_misc: ceph log 'this is a test log message 27256.Sat Feb 3 02:42:56 PM UTC 2024' 2024-02-03T14:42:58.363 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:755: test_mon_misc: ceph log last 2024-02-03T14:42:58.363 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:755: test_mon_misc: grep 'this is a test log message 27256.Sat Feb 3 02:42:56 PM UTC 2024' 2024-02-03T14:42:58.872 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_mon_misc: rm -fr /tmp/cephtool.p0s 2024-02-03T14:42:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:43:58.874 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-02-03T14:44:58.876 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-02-03T14:44:58.877 INFO:tasks.workunit:Stopping ['cephtool'] on client.0... 2024-02-03T14:44:58.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-03T14:44:59.169 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-02-03T14:44:59.378 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f49f43b0d8e04abf9cdea9c2dcd60927 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2024-02-03T14:44:59.381 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-03T14:44:59.391 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-03T14:44:59.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:44:59.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:44:59.695 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:44:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-02-03T14:44:59.262838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":101532,"kb_used_data":3824,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":283013988,"statfs":{"total":289910292480,"available":289806323712,"internally_reserved":0,"allocated":3915776,"data_stored":2574051,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227685+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:42:36.227685+0000","last_peered":"2024-02-03T14:42:36.227685+0000","last_clean":"2024-02-03T14:42:36.227685+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:42:36.227685+0000","last_undegraded":"2024-02-03T14:42:36.227685+0000","last_fullsized":"2024-02-03T14:42:36.227685+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226568+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:42:36.226568+0000","last_peered":"2024-02-03T14:42:36.226568+0000","last_clean":"2024-02-03T14:42:36.226568+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:42:36.226568+0000","last_undegraded":"2024-02-03T14:42:36.226568+0000","last_fullsized":"2024-02-03T14:42:36.226568+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:38.679096+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:42:38.679096+0000","last_peered":"2024-02-03T14:42:38.679096+0000","last_clean":"2024-02-03T14:42:38.679096+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:42:38.679096+0000","last_undegraded":"2024-02-03T14:42:38.679096+0000","last_fullsized":"2024-02-03T14:42:38.679096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227699+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:42:36.227699+0000","last_peered":"2024-02-03T14:42:36.227699+0000","last_clean":"2024-02-03T14:42:36.227699+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:42:36.227699+0000","last_undegraded":"2024-02-03T14:42:36.227699+0000","last_fullsized":"2024-02-03T14:42:36.227699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":263,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226579+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:42:36.226579+0000","last_peered":"2024-02-03T14:42:36.226579+0000","last_clean":"2024-02-03T14:42:36.226579+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:42:36.226579+0000","last_undegraded":"2024-02-03T14:42:36.226579+0000","last_fullsized":"2024-02-03T14:42:36.226579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227789+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:42:36.227789+0000","last_peered":"2024-02-03T14:42:36.227789+0000","last_clean":"2024-02-03T14:42:36.227789+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:42:36.227789+0000","last_undegraded":"2024-02-03T14:42:36.227789+0000","last_fullsized":"2024-02-03T14:42:36.227789+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226695+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:42:36.226695+0000","last_peered":"2024-02-03T14:42:36.226695+0000","last_clean":"2024-02-03T14:42:36.226695+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:42:36.226695+0000","last_undegraded":"2024-02-03T14:42:36.226695+0000","last_fullsized":"2024-02-03T14:42:36.226695+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.228015+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:42:36.228015+0000","last_peered":"2024-02-03T14:42:36.228015+0000","last_clean":"2024-02-03T14:42:36.228015+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:42:36.228015+0000","last_undegraded":"2024-02-03T14:42:36.228015+0000","last_fullsized":"2024-02-03T14:42:36.228015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227931+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:42:36.227931+0000","last_peered":"2024-02-03T14:42:36.227931+0000","last_clean":"2024-02-03T14:42:36.227931+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:42:36.227931+0000","last_undegraded":"2024-02-03T14:42:36.227931+0000","last_fullsized":"2024-02-03T14:42:36.227931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.496,"15min":0.501},"min":{"1min":0.40600000000000003,"5min":0.193,"15min":0.193},"max":{"1min":0.69499999999999995,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.45000000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61199999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.47799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66500000000000004,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.64400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738458,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.439,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":1.71,"15min":1.71},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.52000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71699999999999997,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.437,"5min":0.246,"15min":0.246},"max":{"1min":0.69699999999999995,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.51300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70599999999999996,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.65000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36580,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335260,"statfs":{"total":96636764160,"available":96599306240,"internally_reserved":0,"allocated":1269760,"data_stored":546713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.501,"15min":0.499},"min":{"1min":0.433,"5min":0.216,"15min":0.216},"max":{"1min":0.61599999999999999,"5min":0.89400000000000002,"15min":0.92600000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.45900000000000002,"5min":0.255,"15min":0.23999999999999999},"max":{"1min":0.68100000000000005,"5min":0.876,"15min":0.876},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.49099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.64200000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.48899999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72299999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:44:59.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:44:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:44:59.985 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:45:00.004 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-02-03T14:44:59.262838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":101532,"kb_used_data":3824,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":283013988,"statfs":{"total":289910292480,"available":289806323712,"internally_reserved":0,"allocated":3915776,"data_stored":2574051,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227685+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:42:36.227685+0000","last_peered":"2024-02-03T14:42:36.227685+0000","last_clean":"2024-02-03T14:42:36.227685+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:42:36.227685+0000","last_undegraded":"2024-02-03T14:42:36.227685+0000","last_fullsized":"2024-02-03T14:42:36.227685+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226568+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:42:36.226568+0000","last_peered":"2024-02-03T14:42:36.226568+0000","last_clean":"2024-02-03T14:42:36.226568+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:42:36.226568+0000","last_undegraded":"2024-02-03T14:42:36.226568+0000","last_fullsized":"2024-02-03T14:42:36.226568+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:38.679096+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:42:38.679096+0000","last_peered":"2024-02-03T14:42:38.679096+0000","last_clean":"2024-02-03T14:42:38.679096+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:42:38.679096+0000","last_undegraded":"2024-02-03T14:42:38.679096+0000","last_fullsized":"2024-02-03T14:42:38.679096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227699+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:42:36.227699+0000","last_peered":"2024-02-03T14:42:36.227699+0000","last_clean":"2024-02-03T14:42:36.227699+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:42:36.227699+0000","last_undegraded":"2024-02-03T14:42:36.227699+0000","last_fullsized":"2024-02-03T14:42:36.227699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":263,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226579+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:42:36.226579+0000","last_peered":"2024-02-03T14:42:36.226579+0000","last_clean":"2024-02-03T14:42:36.226579+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:42:36.226579+0000","last_undegraded":"2024-02-03T14:42:36.226579+0000","last_fullsized":"2024-02-03T14:42:36.226579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227789+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:42:36.227789+0000","last_peered":"2024-02-03T14:42:36.227789+0000","last_clean":"2024-02-03T14:42:36.227789+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:42:36.227789+0000","last_undegraded":"2024-02-03T14:42:36.227789+0000","last_fullsized":"2024-02-03T14:42:36.227789+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226695+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:42:36.226695+0000","last_peered":"2024-02-03T14:42:36.226695+0000","last_clean":"2024-02-03T14:42:36.226695+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:42:36.226695+0000","last_undegraded":"2024-02-03T14:42:36.226695+0000","last_fullsized":"2024-02-03T14:42:36.226695+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.228015+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:42:36.228015+0000","last_peered":"2024-02-03T14:42:36.228015+0000","last_clean":"2024-02-03T14:42:36.228015+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:42:36.228015+0000","last_undegraded":"2024-02-03T14:42:36.228015+0000","last_fullsized":"2024-02-03T14:42:36.228015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227931+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:42:36.227931+0000","last_peered":"2024-02-03T14:42:36.227931+0000","last_clean":"2024-02-03T14:42:36.227931+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:42:36.227931+0000","last_undegraded":"2024-02-03T14:42:36.227931+0000","last_fullsized":"2024-02-03T14:42:36.227931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.496,"15min":0.501},"min":{"1min":0.40600000000000003,"5min":0.193,"15min":0.193},"max":{"1min":0.69499999999999995,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.45000000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61199999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.47799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66500000000000004,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.64400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738458,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.439,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":1.71,"15min":1.71},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.52000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71699999999999997,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.437,"5min":0.246,"15min":0.246},"max":{"1min":0.69699999999999995,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.51300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70599999999999996,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.65000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36580,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335260,"statfs":{"total":96636764160,"available":96599306240,"internally_reserved":0,"allocated":1269760,"data_stored":546713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.501,"15min":0.499},"min":{"1min":0.433,"5min":0.216,"15min":0.216},"max":{"1min":0.61599999999999999,"5min":0.89400000000000002,"15min":0.92600000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.45900000000000002,"5min":0.255,"15min":0.23999999999999999},"max":{"1min":0.68100000000000005,"5min":0.876,"15min":0.876},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.49099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.64200000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.48899999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72299999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:45:00.005 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-03T14:45:00.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:45:00.314 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:45:00.314 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:45:00.332 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-02-03T14:44:59.262838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":101532,"kb_used_data":3824,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":283013988,"statfs":{"total":289910292480,"available":289806323712,"internally_reserved":0,"allocated":3915776,"data_stored":2574051,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227685+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:42:36.227685+0000","last_peered":"2024-02-03T14:42:36.227685+0000","last_clean":"2024-02-03T14:42:36.227685+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:42:36.227685+0000","last_undegraded":"2024-02-03T14:42:36.227685+0000","last_fullsized":"2024-02-03T14:42:36.227685+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226568+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:42:36.226568+0000","last_peered":"2024-02-03T14:42:36.226568+0000","last_clean":"2024-02-03T14:42:36.226568+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:42:36.226568+0000","last_undegraded":"2024-02-03T14:42:36.226568+0000","last_fullsized":"2024-02-03T14:42:36.226568+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:38.679096+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:42:38.679096+0000","last_peered":"2024-02-03T14:42:38.679096+0000","last_clean":"2024-02-03T14:42:38.679096+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:42:38.679096+0000","last_undegraded":"2024-02-03T14:42:38.679096+0000","last_fullsized":"2024-02-03T14:42:38.679096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227699+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:42:36.227699+0000","last_peered":"2024-02-03T14:42:36.227699+0000","last_clean":"2024-02-03T14:42:36.227699+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:42:36.227699+0000","last_undegraded":"2024-02-03T14:42:36.227699+0000","last_fullsized":"2024-02-03T14:42:36.227699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":263,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226579+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:42:36.226579+0000","last_peered":"2024-02-03T14:42:36.226579+0000","last_clean":"2024-02-03T14:42:36.226579+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:42:36.226579+0000","last_undegraded":"2024-02-03T14:42:36.226579+0000","last_fullsized":"2024-02-03T14:42:36.226579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227789+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:42:36.227789+0000","last_peered":"2024-02-03T14:42:36.227789+0000","last_clean":"2024-02-03T14:42:36.227789+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:42:36.227789+0000","last_undegraded":"2024-02-03T14:42:36.227789+0000","last_fullsized":"2024-02-03T14:42:36.227789+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226695+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:42:36.226695+0000","last_peered":"2024-02-03T14:42:36.226695+0000","last_clean":"2024-02-03T14:42:36.226695+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:42:36.226695+0000","last_undegraded":"2024-02-03T14:42:36.226695+0000","last_fullsized":"2024-02-03T14:42:36.226695+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.228015+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:42:36.228015+0000","last_peered":"2024-02-03T14:42:36.228015+0000","last_clean":"2024-02-03T14:42:36.228015+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:42:36.228015+0000","last_undegraded":"2024-02-03T14:42:36.228015+0000","last_fullsized":"2024-02-03T14:42:36.228015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227931+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:42:36.227931+0000","last_peered":"2024-02-03T14:42:36.227931+0000","last_clean":"2024-02-03T14:42:36.227931+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:42:36.227931+0000","last_undegraded":"2024-02-03T14:42:36.227931+0000","last_fullsized":"2024-02-03T14:42:36.227931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.496,"15min":0.501},"min":{"1min":0.40600000000000003,"5min":0.193,"15min":0.193},"max":{"1min":0.69499999999999995,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.45000000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61199999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.47799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66500000000000004,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.64400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738458,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.439,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":1.71,"15min":1.71},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.52000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71699999999999997,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.437,"5min":0.246,"15min":0.246},"max":{"1min":0.69699999999999995,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.51300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70599999999999996,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.65000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36580,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335260,"statfs":{"total":96636764160,"available":96599306240,"internally_reserved":0,"allocated":1269760,"data_stored":546713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.501,"15min":0.499},"min":{"1min":0.433,"5min":0.216,"15min":0.216},"max":{"1min":0.61599999999999999,"5min":0.89400000000000002,"15min":0.92600000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.45900000000000002,"5min":0.255,"15min":0.23999999999999999},"max":{"1min":0.68100000000000005,"5min":0.876,"15min":0.876},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.49099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.64200000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.48899999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72299999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:45:00.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-03T14:45:00.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:45:00.625 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":143,"fsid":"2dbfabd7-19a5-49d9-9435-d2e80516e779","created":"2024-02-03T14:37:58.278765+0000","modified":"2024-02-03T14:42:36.215812+0000","last_up_change":"2024-02-03T14:38:06.217130+0000","last_in_change":"2024-02-03T14:37:59.225268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-02-03T14:38:06.885038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-02-03T14:38:07.238989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6eff2159-02fa-462e-937d-98e2359ecd93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6808","nonce":1938254861}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6809","nonce":1938254861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6811","nonce":1938254861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6810","nonce":1938254861}]},"public_addr":"172.21.15.7:6808/1938254861","cluster_addr":"172.21.15.7:6809/1938254861","heartbeat_back_addr":"172.21.15.7:6811/1938254861","heartbeat_front_addr":"172.21.15.7:6810/1938254861","state":["exists","up"]},{"osd":1,"uuid":"428cfc4f-5eb3-4048-b602-43dd7d453c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6800","nonce":2463192558}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6801","nonce":2463192558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6803","nonce":2463192558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6802","nonce":2463192558}]},"public_addr":"172.21.15.7:6800/2463192558","cluster_addr":"172.21.15.7:6801/2463192558","heartbeat_back_addr":"172.21.15.7:6803/2463192558","heartbeat_front_addr":"172.21.15.7:6802/2463192558","state":["exists","up"]},{"osd":2,"uuid":"563c4d0d-9b1e-470d-aaa3-57a4ca92b336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6804","nonce":4029165148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6805","nonce":4029165148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6807","nonce":4029165148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.7:6806","nonce":4029165148}]},"public_addr":"172.21.15.7:6804/4029165148","cluster_addr":"172.21.15.7:6805/4029165148","heartbeat_back_addr":"172.21.15.7:6807/4029165148","heartbeat_front_addr":"172.21.15.7:6806/4029165148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.753654+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.765229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-03T14:38:04.713230+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-03T14:45:01.643 INFO:tasks.ceph:Scrubbing osd.0 2024-02-03T14:45:01.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-02-03T14:45:01.802 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-03T14:45:01.802 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-03T14:45:01.802 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-03T14:45:01.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-02-03T14:45:02.092 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2024-02-03T14:45:02.108 INFO:tasks.ceph:Scrubbing osd.1 2024-02-03T14:45:02.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-02-03T14:45:02.249 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-03T14:45:02.249 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-03T14:45:02.249 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-03T14:45:02.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-02-03T14:45:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2024-02-03T14:45:02.565 INFO:tasks.ceph:Scrubbing osd.2 2024-02-03T14:45:02.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-02-03T14:45:02.708 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-02-03T14:45:02.708 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-03T14:45:02.708 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-02-03T14:45:02.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-02-03T14:45:03.005 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2024-02-03T14:45:03.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:45:03.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:45:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:45:03.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-02-03T14:45:03.264004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":101532,"kb_used_data":3824,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":283013988,"statfs":{"total":289910292480,"available":289806323712,"internally_reserved":0,"allocated":3915776,"data_stored":2574051,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003086"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227685+0000","last_change":"2024-02-03T14:38:08.279819+0000","last_active":"2024-02-03T14:42:36.227685+0000","last_peered":"2024-02-03T14:42:36.227685+0000","last_clean":"2024-02-03T14:42:36.227685+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:42:36.227685+0000","last_undegraded":"2024-02-03T14:42:36.227685+0000","last_fullsized":"2024-02-03T14:42:36.227685+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:38:16.113599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226568+0000","last_change":"2024-02-03T14:38:08.280826+0000","last_active":"2024-02-03T14:42:36.226568+0000","last_peered":"2024-02-03T14:42:36.226568+0000","last_clean":"2024-02-03T14:42:36.226568+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:42:36.226568+0000","last_undegraded":"2024-02-03T14:42:36.226568+0000","last_fullsized":"2024-02-03T14:42:36.226568+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:01:02.828241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:38.679096+0000","last_change":"2024-02-03T14:38:08.284793+0000","last_active":"2024-02-03T14:42:38.679096+0000","last_peered":"2024-02-03T14:42:38.679096+0000","last_clean":"2024-02-03T14:42:38.679096+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:42:38.679096+0000","last_undegraded":"2024-02-03T14:42:38.679096+0000","last_fullsized":"2024-02-03T14:42:38.679096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T19:14:16.841604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227699+0000","last_change":"2024-02-03T14:38:08.281552+0000","last_active":"2024-02-03T14:42:36.227699+0000","last_peered":"2024-02-03T14:42:36.227699+0000","last_clean":"2024-02-03T14:42:36.227699+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:42:36.227699+0000","last_undegraded":"2024-02-03T14:42:36.227699+0000","last_fullsized":"2024-02-03T14:42:36.227699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:01:25.180253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":263,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226579+0000","last_change":"2024-02-03T14:38:08.281706+0000","last_active":"2024-02-03T14:42:36.226579+0000","last_peered":"2024-02-03T14:42:36.226579+0000","last_clean":"2024-02-03T14:42:36.226579+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:42:36.226579+0000","last_undegraded":"2024-02-03T14:42:36.226579+0000","last_fullsized":"2024-02-03T14:42:36.226579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T23:08:58.017155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227789+0000","last_change":"2024-02-03T14:38:08.284531+0000","last_active":"2024-02-03T14:42:36.227789+0000","last_peered":"2024-02-03T14:42:36.227789+0000","last_clean":"2024-02-03T14:42:36.227789+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:42:36.227789+0000","last_undegraded":"2024-02-03T14:42:36.227789+0000","last_fullsized":"2024-02-03T14:42:36.227789+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T14:53:09.981795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.226695+0000","last_change":"2024-02-03T14:38:08.283182+0000","last_active":"2024-02-03T14:42:36.226695+0000","last_peered":"2024-02-03T14:42:36.226695+0000","last_clean":"2024-02-03T14:42:36.226695+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:42:36.226695+0000","last_undegraded":"2024-02-03T14:42:36.226695+0000","last_fullsized":"2024-02-03T14:42:36.226695+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:34:27.113820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.228015+0000","last_change":"2024-02-03T14:38:08.282915+0000","last_active":"2024-02-03T14:42:36.228015+0000","last_peered":"2024-02-03T14:42:36.228015+0000","last_clean":"2024-02-03T14:42:36.228015+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:42:36.228015+0000","last_undegraded":"2024-02-03T14:42:36.228015+0000","last_fullsized":"2024-02-03T14:42:36.228015+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:44:20.305606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:42:36.227931+0000","last_change":"2024-02-03T14:38:08.281448+0000","last_active":"2024-02-03T14:42:36.227931+0000","last_peered":"2024-02-03T14:42:36.227931+0000","last_clean":"2024-02-03T14:42:36.227931+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:42:36.227931+0000","last_undegraded":"2024-02-03T14:42:36.227931+0000","last_fullsized":"2024-02-03T14:42:36.227931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:38:07.262531+0000","last_clean_scrub_stamp":"2024-02-03T14:38:07.262531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T16:12:49.666997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738459,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.496,"15min":0.501},"min":{"1min":0.40600000000000003,"5min":0.193,"15min":0.193},"max":{"1min":0.69499999999999995,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.45000000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61199999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.47799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66500000000000004,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.502},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.64400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738459,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.439,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":1.71,"15min":1.71},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.52000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71699999999999997,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.504}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.437,"5min":0.246,"15min":0.246},"max":{"1min":0.69699999999999995,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.51300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70599999999999996,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.54800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36580,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335260,"statfs":{"total":96636764160,"available":96599306240,"internally_reserved":0,"allocated":1269760,"data_stored":546713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.501,"15min":0.499},"min":{"1min":0.433,"5min":0.216,"15min":0.216},"max":{"1min":0.61599999999999999,"5min":0.89400000000000002,"15min":0.92600000000000005},"last":0.436},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.45900000000000002,"5min":0.255,"15min":0.23999999999999999},"max":{"1min":0.68100000000000005,"5min":0.876,"15min":0.876},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.49099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.64200000000000002,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.48899999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72299999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:45:03.322 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.322 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.322 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.322 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.323 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.323 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.323 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.323 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.323 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-02-03T14:38:07.262531+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=38, tm_sec=7, tm_wday=5, tm_yday=34, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=3, tm_hour=14, tm_min=45, tm_sec=0, tm_wday=5, tm_yday=34, tm_isdst=0) 2024-02-03T14:45:03.324 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-02-03T14:45:23.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-03T14:45:23.626 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-02-03T14:45:23.626 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-02-03T14:45:23.644 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-02-03T14:45:23.268939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":101512,"kb_used_data":3824,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":283014008,"statfs":{"total":289910292480,"available":289806344192,"internally_reserved":0,"allocated":3915776,"data_stored":2574051,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002930"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:02.772536+0000","last_change":"2024-02-03T14:45:02.772555+0000","last_active":"2024-02-03T14:45:02.772536+0000","last_peered":"2024-02-03T14:45:02.772536+0000","last_clean":"2024-02-03T14:45:02.772536+0000","last_became_active":"2024-02-03T14:38:08.279375+0000","last_became_peered":"2024-02-03T14:38:08.279375+0000","last_unstale":"2024-02-03T14:45:02.772536+0000","last_undegraded":"2024-02-03T14:45:02.772536+0000","last_fullsized":"2024-02-03T14:45:02.772536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:02.772459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:02.772459+0000","last_clean_scrub_stamp":"2024-02-03T14:45:02.772459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T18:25:30.685247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":145,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:05.852706+0000","last_change":"2024-02-03T14:45:05.852753+0000","last_active":"2024-02-03T14:45:05.852706+0000","last_peered":"2024-02-03T14:45:05.852706+0000","last_clean":"2024-02-03T14:45:05.852706+0000","last_became_active":"2024-02-03T14:38:08.280530+0000","last_became_peered":"2024-02-03T14:38:08.280530+0000","last_unstale":"2024-02-03T14:45:05.852706+0000","last_undegraded":"2024-02-03T14:45:05.852706+0000","last_fullsized":"2024-02-03T14:45:05.852706+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:05.852541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:05.852541+0000","last_clean_scrub_stamp":"2024-02-03T14:45:05.852541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T01:16:27.609649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":140,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:02.627002+0000","last_change":"2024-02-03T14:45:02.627030+0000","last_active":"2024-02-03T14:45:02.627002+0000","last_peered":"2024-02-03T14:45:02.627002+0000","last_clean":"2024-02-03T14:45:02.627002+0000","last_became_active":"2024-02-03T14:38:08.284366+0000","last_became_peered":"2024-02-03T14:38:08.284366+0000","last_unstale":"2024-02-03T14:45:02.627002+0000","last_undegraded":"2024-02-03T14:45:02.627002+0000","last_fullsized":"2024-02-03T14:45:02.627002+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:02.626879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:02.626879+0000","last_clean_scrub_stamp":"2024-02-03T14:45:02.626879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:57:07.577549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:05.781728+0000","last_change":"2024-02-03T14:45:05.781775+0000","last_active":"2024-02-03T14:45:05.781728+0000","last_peered":"2024-02-03T14:45:05.781728+0000","last_clean":"2024-02-03T14:45:05.781728+0000","last_became_active":"2024-02-03T14:38:08.281097+0000","last_became_peered":"2024-02-03T14:38:08.281097+0000","last_unstale":"2024-02-03T14:45:05.781728+0000","last_undegraded":"2024-02-03T14:45:05.781728+0000","last_fullsized":"2024-02-03T14:45:05.781728+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:05.781509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:05.781509+0000","last_clean_scrub_stamp":"2024-02-03T14:45:05.781509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:36:34.492595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"143'99","reported_seq":274,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:03.863199+0000","last_change":"2024-02-03T14:45:03.863270+0000","last_active":"2024-02-03T14:45:03.863199+0000","last_peered":"2024-02-03T14:45:03.863199+0000","last_clean":"2024-02-03T14:45:03.863199+0000","last_became_active":"2024-02-03T14:38:08.281444+0000","last_became_peered":"2024-02-03T14:38:08.281444+0000","last_unstale":"2024-02-03T14:45:03.863199+0000","last_undegraded":"2024-02-03T14:45:03.863199+0000","last_fullsized":"2024-02-03T14:45:03.863199+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'99","last_scrub_stamp":"2024-02-03T14:45:03.863016+0000","last_deep_scrub":"143'99","last_deep_scrub_stamp":"2024-02-03T14:45:03.863016+0000","last_clean_scrub_stamp":"2024-02-03T14:45:03.863016+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T01:59:53.555325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:04.816483+0000","last_change":"2024-02-03T14:45:04.816534+0000","last_active":"2024-02-03T14:45:04.816483+0000","last_peered":"2024-02-03T14:45:04.816483+0000","last_clean":"2024-02-03T14:45:04.816483+0000","last_became_active":"2024-02-03T14:38:08.284132+0000","last_became_peered":"2024-02-03T14:38:08.284132+0000","last_unstale":"2024-02-03T14:45:04.816483+0000","last_undegraded":"2024-02-03T14:45:04.816483+0000","last_fullsized":"2024-02-03T14:45:04.816483+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:04.816303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:04.816303+0000","last_clean_scrub_stamp":"2024-02-03T14:45:04.816303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T21:47:42.225063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":145,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:04.819191+0000","last_change":"2024-02-03T14:45:04.819241+0000","last_active":"2024-02-03T14:45:04.819191+0000","last_peered":"2024-02-03T14:45:04.819191+0000","last_clean":"2024-02-03T14:45:04.819191+0000","last_became_active":"2024-02-03T14:38:08.282883+0000","last_became_peered":"2024-02-03T14:38:08.282883+0000","last_unstale":"2024-02-03T14:45:04.819191+0000","last_undegraded":"2024-02-03T14:45:04.819191+0000","last_fullsized":"2024-02-03T14:45:04.819191+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:04.819033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:04.819033+0000","last_clean_scrub_stamp":"2024-02-03T14:45:04.819033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T15:20:38.970685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:06.795342+0000","last_change":"2024-02-03T14:45:06.795392+0000","last_active":"2024-02-03T14:45:06.795342+0000","last_peered":"2024-02-03T14:45:06.795342+0000","last_clean":"2024-02-03T14:45:06.795342+0000","last_became_active":"2024-02-03T14:38:08.282389+0000","last_became_peered":"2024-02-03T14:38:08.282389+0000","last_unstale":"2024-02-03T14:45:06.795342+0000","last_undegraded":"2024-02-03T14:45:06.795342+0000","last_fullsized":"2024-02-03T14:45:06.795342+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:06.795166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:06.795166+0000","last_clean_scrub_stamp":"2024-02-03T14:45:06.795166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T17:39:38.923705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-02-03T14:45:03.793603+0000","last_change":"2024-02-03T14:45:03.793678+0000","last_active":"2024-02-03T14:45:03.793603+0000","last_peered":"2024-02-03T14:45:03.793603+0000","last_clean":"2024-02-03T14:45:03.793603+0000","last_became_active":"2024-02-03T14:38:08.281118+0000","last_became_peered":"2024-02-03T14:38:08.281118+0000","last_unstale":"2024-02-03T14:45:03.793603+0000","last_undegraded":"2024-02-03T14:45:03.793603+0000","last_fullsized":"2024-02-03T14:45:03.793603+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-03T14:45:03.793408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-03T14:45:03.793408+0000","last_clean_scrub_stamp":"2024-02-03T14:45:03.793408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-04T21:14:46.450899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48948,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738463,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28308,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343532,"statfs":{"total":96636764160,"available":96607776768,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.496,"15min":0.501},"min":{"1min":0.40600000000000003,"5min":0.193,"15min":0.193},"max":{"1min":0.69499999999999995,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.45000000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61199999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Sat Feb 3 14:44:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.47799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66500000000000004,"5min":3.0179999999999998,"15min":3.0179999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47699999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77500000000000002,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.55000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738463,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1323008,"data_stored":1013669,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 3 14:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53900000000000003,"15min":0.499},"min":{"1min":0.44500000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67500000000000004,"5min":1.71,"15min":1.71},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.504,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.74299999999999999,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Feb 3 14:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.54500000000000004,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68400000000000005,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.65800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36576,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335264,"statfs":{"total":96636764160,"available":96599310336,"internally_reserved":0,"allocated":1269760,"data_stored":546713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 3 14:45:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.40600000000000003,"5min":0.216,"15min":0.216},"max":{"1min":0.68300000000000005,"5min":0.73499999999999999,"15min":0.92600000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42999999999999999,"5min":0.255,"15min":0.23999999999999999},"max":{"1min":0.61799999999999999,"5min":0.876,"15min":0.876},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Feb 3 14:45:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.47699999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66300000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.503},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58199999999999996,"15min":0.55900000000000005},"min":{"1min":0.47199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68100000000000005,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24474,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-03T14:45:23.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-02-03T14:45:23.957 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-03T14:45:23.957 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-03T14:45:23.957 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-02-03T14:45:23.957 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:24.018 INFO:tasks.ceph.osd.0:Stopped 2024-02-03T14:45:24.018 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-02-03T14:45:24.018 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:28.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-02-03T14:45:30.120 INFO:tasks.ceph.osd.1:Stopped 2024-02-03T14:45:30.120 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-02-03T14:45:30.120 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:30.177 INFO:tasks.ceph.osd.2:Stopped 2024-02-03T14:45:30.178 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-03T14:45:30.178 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-02-03T14:45:30.178 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:30.209 INFO:tasks.ceph.mgr.x:Stopped 2024-02-03T14:45:30.209 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-03T14:45:30.209 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-02-03T14:45:30.209 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:30.264 INFO:tasks.ceph.mon.a:Stopped 2024-02-03T14:45:30.264 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-02-03T14:45:30.264 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:30.319 INFO:tasks.ceph.mon.b:Stopped 2024-02-03T14:45:30.322 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-02-03T14:45:30.323 INFO:teuthology.orchestra.run:waiting for 300 2024-02-03T14:45:30.376 INFO:tasks.ceph.mon.c:Stopped 2024-02-03T14:45:30.377 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-03T14:45:30.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-03T14:45:30.435 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:45:30.436 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-02-03T14:45:30.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:45:30.666 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-02-03T14:45:30.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:45:30.865 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-02-03T14:45:31.190 INFO:tasks.ceph:Archiving mon data... 2024-02-03T14:45:31.191 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/data/mon.a.tgz 2024-02-03T14:45:31.191 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-02-03T14:45:31.197 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.MX7cvOgMdn 2024-02-03T14:45:31.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MX7cvOgMdn 2024-02-03T14:45:31.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.MX7cvOgMdn 2024-02-03T14:45:31.384 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.MX7cvOgMdn is 288KB 2024-02-03T14:45:31.445 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.MX7cvOgMdn 2024-02-03T14:45:31.451 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/data/mon.b.tgz 2024-02-03T14:45:31.451 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-02-03T14:45:31.499 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.G4A6npvjxN 2024-02-03T14:45:31.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.G4A6npvjxN 2024-02-03T14:45:31.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.G4A6npvjxN 2024-02-03T14:45:31.688 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.G4A6npvjxN is 289KB 2024-02-03T14:45:31.745 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.G4A6npvjxN 2024-02-03T14:45:31.750 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/data/mon.c.tgz 2024-02-03T14:45:31.750 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-02-03T14:45:31.795 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.6EQgoB1MGx 2024-02-03T14:45:31.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6EQgoB1MGx 2024-02-03T14:45:31.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.6EQgoB1MGx 2024-02-03T14:45:31.975 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.6EQgoB1MGx is 289KB 2024-02-03T14:45:32.033 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.6EQgoB1MGx 2024-02-03T14:45:32.038 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-03T14:45:32.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-03T14:45:32.140 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-03T14:45:32.140 INFO:tasks.ceph:Archiving crash dumps... 2024-02-03T14:45:32.142 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/remote/smithi007/crash 2024-02-03T14:45:32.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-03T14:45:32.196 INFO:tasks.ceph:Compressing logs... 2024-02-03T14:45:32.196 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-03T14:45:32.267 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28775.log 2024-02-03T14:45:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27831.log 2024-02-03T14:45:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28775.log.gz 2024-02-03T14:45:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29448.log 2024-02-03T14:45:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33750.log 2024-02-03T14:45:32.269 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27831.log.gz 2024-02-03T14:45:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29448.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.34810.log -- replaced with /var/log/ceph/ceph-client.admin.29448.log.gz 2024-02-03T14:45:32.270 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5/var/log/ceph/ceph-client.xx-profile-ro.33750.log: --verbose -- /var/log/ceph/ceph-client.admin.26658.log 2024-02-03T14:45:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33750.log.gz 2024-02-03T14:45:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz -- 2024-02-03T14:45:32.271 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.xx-profile-rw.33963.log 2024-02-03T14:45:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26658.log.gz 2024-02-03T14:45:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28451.log 2024-02-03T14:45:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.33963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30678.log 2024-02-03T14:45:32.273 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.33963.log.gz 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28451.log.gz 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30143.log 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32973.log 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30678.log.gz 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28539.log 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30143.log.gz 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd2.34484.log 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32973.log.gz 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31333.log 2024-02-03T14:45:32.274 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28539.log.gz 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd2.34484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26785.log 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd2.34484.log.gz 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34178.log 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31333.log.gz 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26785.log.gz 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28074.log 2024-02-03T14:45:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27024.log 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34178.log.gz 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26789.log 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28074.log.gz 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27024.log.gz 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32697.log 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26789.log.gz 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28657.log 2024-02-03T14:45:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32697.log.gz 2024-02-03T14:45:32.280 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-02-03T14:45:32.280 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27636.log 2024-02-03T14:45:32.280 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35026.log 2024-02-03T14:45:32.280 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28657.log.gz 2024-02-03T14:45:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32393.log 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27636.log.gz 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35026.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35026.log.gz 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30191.log 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26945.log 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32393.log.gz 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32774.log 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30191.log.gz 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26945.log.gz 2024-02-03T14:45:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30487.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27538.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32774.log.gz 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30487.log.gz 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27538.log.gz 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27366.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34342.log 2024-02-03T14:45:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27366.log.gz 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29752.log 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27783.log 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34342.log.gz 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34786.log 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29752.log.gz 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29518.log 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27783.log.gz 2024-02-03T14:45:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34412.log 2024-02-03T14:45:32.289 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34786.log.gz 2024-02-03T14:45:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29518.log.gz 2024-02-03T14:45:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33115.log 2024-02-03T14:45:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34608.log 2024-02-03T14:45:32.289 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34412.log.gz 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30581.log 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33115.log.gz 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34608.log.gz 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-02-03T14:45:32.290 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30581.log.gz 2024-02-03T14:45:32.311 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27685.log 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31312.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz --verbose 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.31121.log 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35349.log 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27685.log.gz 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35297.log 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31121.log.gz 2024-02-03T14:45:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30923.log 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35349.log.gz 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29472.log 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35297.log.gz 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30923.log.gz 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28325.log 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29472.log.gz 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30823.log 2024-02-03T14:45:32.313 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29776.log 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28325.log.gz 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27120.log 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30823.log.gz 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29776.log.gz 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30338.log 2024-02-03T14:45:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27120.log.gz 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31816.log 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23785.log 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30338.log.gz 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31866.log 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31816.log.gz 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33774.log 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23785.log.gz 2024-02-03T14:45:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31866.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31866.log.gz -5 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30728.log 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32033.log 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33774.log.gz 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30728.log.gz 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30386.log 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31551.log 0.0% 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32033.log.gz 2024-02-03T14:45:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23930.log 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30386.log.gz 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31551.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30704.log 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31551.log.gz 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23930.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23930.log.gz 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32172.log 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30704.log.gz 2024-02-03T14:45:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.34059.log 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32172.log.gz 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5/var/log/ceph/ceph-client.admin.34684.log: --verbose -- /var/log/ceph/ceph-client.admin.27661.log 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.34059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.34059.log.gz 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-02-03T14:45:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27661.log.gz 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32851.log 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28609.log 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32851.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32851.log.gz 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28633.log 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28609.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.19993.log 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28609.log.gz 2024-02-03T14:45:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33607.log 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28633.log.gz 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.19993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30242.log 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.19993.log.gz 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33607.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33607.log.gz -5 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31717.log 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30242.log.gz 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29845.log 2024-02-03T14:45:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31717.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29845.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.26860.log 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29845.log.gz 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32925.log 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26860.log.gz 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33629.log 2024-02-03T14:45:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33140.log 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32925.log.gz 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30095.log 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33629.log.gz 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33140.log.gz 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31450.log 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26578.log 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30095.log.gz 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31450.log.gz 2024-02-03T14:45:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27734.log 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26578.log.gz 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29267.log 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27734.log.gz 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34533.log 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29267.log.gz 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31072.log 2024-02-03T14:45:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33020.log 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34533.log.gz 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28399.log 0.0% 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31072.log.gz 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33020.log.gz 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29706.log 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28399.log: /var/log/ceph/ceph-client.admin.33191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28399.log.gz 2024-02-03T14:45:32.324 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31378.log 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27341.log 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29706.log.gz 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32901.log 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31378.log.gz 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27341.log.gz 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28493.log 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28100.log 2024-02-03T14:45:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32901.log.gz 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28493.log.gz 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31575.log 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28100.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28100.log.gz 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29075.log 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28585.log 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31575.log.gz 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32417.log 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29075.log.gz 2024-02-03T14:45:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28585.log.gz 2024-02-03T14:45:32.327 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32573.log 2024-02-03T14:45:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28151.log 2024-02-03T14:45:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32417.log: /var/log/ceph/ceph-client.admin.32573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32417.log.gz 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32573.log.gz 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35323.log 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31172.log 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28151.log.gz 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34272.log 0.0% 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35323.log.gz 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28373.log 2024-02-03T14:45:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31172.log.gz 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32799.log 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34272.log.gz 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28373.log.gz 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29995.log 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32799.log.gz 2024-02-03T14:45:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.33915.log 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28125.log 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29995.log.gz 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.33915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27439.log 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.33915.log.gz 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33289.log 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28125.log.gz 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27439.log.gz 2024-02-03T14:45:32.373 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30557.log 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31023.log 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33289.log.gz 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30557.log.gz 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33654.log 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29586.log 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31023.log.gz 2024-02-03T14:45:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33654.log.gz 2024-02-03T14:45:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34978.log 2024-02-03T14:45:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31693.log 2024-02-03T14:45:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29586.log.gz 2024-02-03T14:45:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34978.log.gz 2024-02-03T14:45:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34761.log 2024-02-03T14:45:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28277.log 2024-02-03T14:45:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31693.log.gz 2024-02-03T14:45:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30411.log 2024-02-03T14:45:32.377 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34761.log.gz 2024-02-03T14:45:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35431.log 2024-02-03T14:45:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28277.log.gz 2024-02-03T14:45:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35050.log 2024-02-03T14:45:32.378 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30411.log.gz 2024-02-03T14:45:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32998.log 2024-02-03T14:45:32.378 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35431.log.gz 2024-02-03T14:45:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29872.log 2024-02-03T14:45:32.379 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35050.log.gz 2024-02-03T14:45:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32998.log.gz 2024-02-03T14:45:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-02-03T14:45:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32009.log 2024-02-03T14:45:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29872.log.gz 2024-02-03T14:45:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29400.log 2024-02-03T14:45:32.380 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-02-03T14:45:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32009.log.gz 2024-02-03T14:45:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33678.log 2024-02-03T14:45:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26603.log 2024-02-03T14:45:32.381 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29400.log.gz 2024-02-03T14:45:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29189.log 2024-02-03T14:45:32.382 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33678.log.gz 2024-02-03T14:45:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26603.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26603.log.gz -5 2024-02-03T14:45:32.382 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33340.log 2024-02-03T14:45:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29144.log 2024-02-03T14:45:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29189.log.gz 2024-02-03T14:45:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31598.log 2024-02-03T14:45:32.383 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33340.log.gz 2024-02-03T14:45:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29144.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29144.log.gz 2024-02-03T14:45:32.384 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/ceph-client.admin.34633.log 2024-02-03T14:45:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-02-03T14:45:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31598.log.gz 2024-02-03T14:45:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28890.log 2024-02-03T14:45:32.387 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2024-02-03T14:45:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27316.log 2024-02-03T14:45:32.388 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-02-03T14:45:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29728.log 2024-02-03T14:45:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28890.log.gz 2024-02-03T14:45:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29123.log 2024-02-03T14:45:32.389 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27316.log.gz 2024-02-03T14:45:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30461.log 2024-02-03T14:45:32.389 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29728.log.gz 2024-02-03T14:45:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27291.log 2024-02-03T14:45:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29123.log.gz 2024-02-03T14:45:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30605.log 2024-02-03T14:45:32.390 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30461.log.gz 2024-02-03T14:45:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33726.log 2024-02-03T14:45:32.390 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27291.log.gz 2024-02-03T14:45:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29165.log 2024-02-03T14:45:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30605.log.gz 2024-02-03T14:45:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.34035.log 2024-02-03T14:45:32.391 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33726.log.gz 2024-02-03T14:45:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32443.log 2024-02-03T14:45:32.392 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29165.log.gz 2024-02-03T14:45:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26712.log 2024-02-03T14:45:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.34035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.34035.log.gz 2024-02-03T14:45:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28753.log 2024-02-03T14:45:32.393 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32443.log.gz 2024-02-03T14:45:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27611.log 2024-02-03T14:45:32.393 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26712.log.gz 2024-02-03T14:45:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30266.log/var/log/ceph/ceph-client.admin.28753.log: 2024-02-03T14:45:32.394 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28753.log.gz 2024-02-03T14:45:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27611.log.gz 2024-02-03T14:45:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-02-03T14:45:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34906.log 2024-02-03T14:45:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30266.log.gz 2024-02-03T14:45:32.395 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32621.log 2024-02-03T14:45:32.395 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-02-03T14:45:32.396 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34460.log 2024-02-03T14:45:32.396 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-02-03T14:45:32.396 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30070.log 2024-02-03T14:45:32.397 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32621.log.gz 2024-02-03T14:45:32.397 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30750.log 2024-02-03T14:45:32.397 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34460.log.gz 2024-02-03T14:45:32.397 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29237.log 2024-02-03T14:45:32.398 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30070.log.gz 2024-02-03T14:45:32.398 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27587.log 2024-02-03T14:45:32.398 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30750.log.gz 2024-02-03T14:45:32.398 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30774.log 2024-02-03T14:45:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29237.log.gz 2024-02-03T14:45:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27489.log 2024-02-03T14:45:32.399 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27587.log.gz 2024-02-03T14:45:32.399 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30774.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30774.log.gz --verbose 2024-02-03T14:45:32.400 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.28818.log 2024-02-03T14:45:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-02-03T14:45:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27489.log.gz 2024-02-03T14:45:32.400 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29322.log 2024-02-03T14:45:32.400 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28818.log.gz 2024-02-03T14:45:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.33939.log 2024-02-03T14:45:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31792.log 2024-02-03T14:45:32.401 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29322.log.gz 2024-02-03T14:45:32.401 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.33939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30119.log 2024-02-03T14:45:32.402 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.33939.log.gz 2024-02-03T14:45:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28975.log 2024-02-03T14:45:32.402 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31792.log.gz 2024-02-03T14:45:32.402 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27759.log 2024-02-03T14:45:32.403 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30119.log.gz 2024-02-03T14:45:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34583.log 2024-02-03T14:45:32.403 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28975.log.gz 2024-02-03T14:45:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26790.log 2024-02-03T14:45:32.403 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27759.log.gz 2024-02-03T14:45:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31963.log 2024-02-03T14:45:32.404 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34583.log.gz 2024-02-03T14:45:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33798.log 2024-02-03T14:45:32.404 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26790.log.gz 2024-02-03T14:45:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35401.log 2024-02-03T14:45:32.405 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31963.log.gz 2024-02-03T14:45:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29494.log 2024-02-03T14:45:32.405 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33798.log.gz 2024-02-03T14:45:32.405 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30656.log 2024-02-03T14:45:32.405 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35401.log.gz 2024-02-03T14:45:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28052.log 2024-02-03T14:45:32.406 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29494.log.gz 2024-02-03T14:45:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32672.log 2024-02-03T14:45:32.406 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30656.log.gz 2024-02-03T14:45:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33439.log 2024-02-03T14:45:32.407 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28052.log.gz 2024-02-03T14:45:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34930.log 2024-02-03T14:45:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32672.log.gz 2024-02-03T14:45:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29352.log 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33439.log.gz 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.34930.log: /var/log/ceph/ceph-client.xx-profile-rw.34107.log 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34930.log.gz 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32196.log 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29352.log.gz 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.34107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33846.log 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.34107.log.gz 2024-02-03T14:45:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32196.log.gz 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32521.log 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33846.log.gz 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29213.log 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33390.log 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32521.log.gz 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33822.log 2024-02-03T14:45:32.409 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29213.log.gz 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28561.log 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33390.log.gz 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30216.log 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33822.log.gz 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32270.log 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28561.log.gz 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27854.log 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30216.log.gz 2024-02-03T14:45:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26867.log 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32270.log.gz 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32087.log 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27854.log.gz 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28225.log 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26867.log.gz 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35245.log 2024-02-03T14:45:32.411 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32087.log.gz 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28225.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28225.log.gz 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30045.log 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35245.log.gz 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32342.log 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30952.log 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30045.log.gz 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28729.log 2024-02-03T14:45:32.412 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32342.log.gz 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30952.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30952.log.gz 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33702.log 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28729.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31354.log 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28729.log.gz 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33702.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27266.log 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33702.log.gz 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31354.log.gz 2024-02-03T14:45:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32112.log 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27464.log 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27266.log.gz 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32112.log.gz 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30873.log 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr: 94.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34659.log 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27464.log.gz 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30873.log.gz 2024-02-03T14:45:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31645.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34659.log: /var/log/ceph/ceph-client.admin.30292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34659.log.gzgzip 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33536.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31645.log.gz 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31222.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34834.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33536.log.gz 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28251.log 2024-02-03T14:45:32.415 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31222.log.gz 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29947.log 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34834.log: 0.0%/var/log/ceph/ceph-client.admin.28251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27808.log 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34834.log.gz 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28251.log.gz 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29947.log.gz 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27710.log 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31291.log 2024-02-03T14:45:32.416 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27808.log.gz 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31942.log 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27710.log.gz 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31291.log.gz 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27390.log 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31942.log: /var/log/ceph/ceph-client.admin.35455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31942.log.gz 2024-02-03T14:45:32.417 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-02-03T14:45:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33314.log 2024-02-03T14:45:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26756.log 2024-02-03T14:45:32.418 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27390.log.gz 2024-02-03T14:45:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34388.log 2024-02-03T14:45:32.419 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33314.log.gz 2024-02-03T14:45:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26756.log.gz 2024-02-03T14:45:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33869.log 2024-02-03T14:45:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34736.log 2024-02-03T14:45:32.420 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34388.log.gz 2024-02-03T14:45:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29925.log 2024-02-03T14:45:32.420 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33869.log.gz 2024-02-03T14:45:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-02-03T14:45:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31146.log 2024-02-03T14:45:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34736.log.gz 2024-02-03T14:45:32.421 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29925.log.gz 2024-02-03T14:45:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23613.log 2024-02-03T14:45:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32949.log 2024-02-03T14:45:32.422 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31146.log.gz 2024-02-03T14:45:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23613.log.gz 2024-02-03T14:45:32.423 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29682.log 2024-02-03T14:45:32.423 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29971.log 2024-02-03T14:45:32.423 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32949.log.gz 2024-02-03T14:45:32.423 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29682.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33366.log 2024-02-03T14:45:32.424 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29682.log.gz 2024-02-03T14:45:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29971.log.gz 2024-02-03T14:45:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28866.log 2024-02-03T14:45:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26946.log 2024-02-03T14:45:32.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33366.log: /var/log/ceph/ceph-client.admin.28866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33366.log.gz 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28866.log.gz 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32825.log 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26946.log.gz 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35104.log 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27020.log 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32825.log.gz 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35104.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35104.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.26868.log 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27020.log.gz 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33489.log 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28945.log 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26868.log.gz 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33265.log 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33489.log.gz 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28945.log.gz 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34882.log 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-02-03T14:45:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33265.log.gz 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34882.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29542.log 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29097.log 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29542.log.gz 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34319.log 2024-02-03T14:45:32.471 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29097.log.gz 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34130.log 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29292.log 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34319.log.gz 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34130.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34249.log 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34130.log.gz 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29292.log.gz 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30798.log 2024-02-03T14:45:32.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28842.log 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34249.log.gz 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30798.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28201.log 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28842.log.gz 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28472.log 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28201.log.gz 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27513.log 2024-02-03T14:45:32.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28472.log.gz 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27513.log.gz 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29564.log 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32246.log 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29564.log.gz 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29658.log 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32246.log.gz 2024-02-03T14:45:32.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34202.log 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29658.log.gz 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30899.log 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34202.log.gz 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23809.log 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30899.log.gz 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30999.log 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr: 95.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28517.log 2024-02-03T14:45:32.475 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23809.log.gz 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30999.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30975.log 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30999.log.gz 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28517.log.gz 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33090.log 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32367.log 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30975.log.gz 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33090.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32597.log 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.476 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32367.log.gz 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29800.log 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32294.log 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29800.log: /var/log/ceph/ceph-client.admin.32597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32597.log.gz 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29800.log.gz 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27944.log 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26631.log 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32294.log.gz 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27944.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29634.log 2024-02-03T14:45:32.477 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26631.log.gz 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32057.log 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29634.log.gz 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34508.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31621.log 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32057.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26938.log 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32057.log.gz 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31621.log.gz 2024-02-03T14:45:32.478 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31097.log 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.26938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31270.log 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26938.log.gz 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31097.log.gz 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31742.log 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33066.log 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31270.log.gz 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31742.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31742.log.gz 2024-02-03T14:45:32.479 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33066.log.gz 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28347.log 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28681.log 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28347.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31669.log 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28681.log.gz 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27980.log 2024-02-03T14:45:32.480 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32646.log 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31669.log.gz 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27980.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32646.log.gz 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28797.log 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28175.log 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28797.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34226.log 2024-02-03T14:45:32.481 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28797.log.gz 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28175.log.gz 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32222.log 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31525.log 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34226.log.gz 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32222.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27146.log 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31525.log.gz 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32495.log 2024-02-03T14:45:32.482 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23868.log 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32495.log: /var/log/ceph/ceph-client.admin.27146.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27146.log.gz 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32495.log.gz 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34296.log 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33241.log 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23868.log.gz 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34296.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34954.log 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34296.log.gz 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33241.log.gz 2024-02-03T14:45:32.483 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29899.log 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34154.log 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34954.log.gz 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29899.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29899.log.gz 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rd.34154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34154.log.gz 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.33987.log 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31246.log 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.xx-profile-rw.33987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.33987.log.gz 2024-02-03T14:45:32.484 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23833.log 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31246.log.gz 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32748.log 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.23833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23833.log.gz 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29053.log 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32748.log.gz 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30167.log 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29053.log.gz 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34858.log 2024-02-03T14:45:32.485 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30167.log.gz 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29610.log 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.34858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34858.log.gz 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29824.log 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29610.log.gz 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30849.log 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29824.log.gz 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28920.log 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30849.log.gz 2024-02-03T14:45:32.486 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32142.log 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28920.log.gz 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28425.log 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32142.log.gz 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27094.log 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28425.log.gz 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.33892.log 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27094.log.gz 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34436.log 2024-02-03T14:45:32.487 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29376.log 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-ro.33892.log: /var/log/ceph/ceph-client.xx-profile-rd.34436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.33892.log.gz 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34436.log.gz 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31402.log 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29376.log.gz 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29005.log 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31842.log 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31402.log.gz 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30020.log 2024-02-03T14:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29005.log.gz 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31842.log.gz 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.34083.log 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33584.log 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30020.log: /var/log/ceph/ceph-client.xx-profile-rw.34083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30020.log.gz 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.34083.log.gz 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33463.log 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33584.log: /var/log/ceph/ceph-client.admin.33463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33584.log.gz 2024-02-03T14:45:32.489 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33463.log.gz 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30511.log 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32877.log 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30511.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30533.log 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30511.log.gz 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32877.log.gz 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32468.log 2024-02-03T14:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28301.log 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30533.log: /var/log/ceph/ceph-client.admin.32468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30533.log.gz 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32468.log.gz 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31501.log 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.34011.log 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28301.log: /var/log/ceph/ceph-client.admin.31501.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31501.log.gz -- replaced with /var/log/ceph/ceph-client.admin.28301.log.gz 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35375.log 2024-02-03T14:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.xx-profile-rw.34011.log: /var/log/ceph/ceph-client.admin.34710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.34011.log.gz 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27016.log 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32547.log 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35375.log: /var/log/ceph/ceph-client.admin.27016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35375.log.gz 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27016.log.gz 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32318.log 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32723.log 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.32547.log: /var/log/ceph/ceph-client.admin.32318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32547.log.gz 2024-02-03T14:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32318.log.gz 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31985.log 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28016.log 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31985.log: /var/log/ceph/ceph-client.admin.32723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32723.log.gz 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31985.log.gz 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33559.log 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31766.log 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.28016.log: /var/log/ceph/ceph-client.admin.33559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28016.log.gz 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33559.log.gz 2024-02-03T14:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.34365.log 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31196.log 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31766.log: /var/log/ceph/ceph-client.xx-profile-rd.34365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31766.log.gz 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.34365.log.gz 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31426.log 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27562.log 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31196.log: /var/log/ceph/ceph-client.admin.31426.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31196.log.gz 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31426.log.gz 2024-02-03T14:45:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35002.log 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29424.log 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27562.log.gz 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35002.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33415.log 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35002.log.gz 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29424.log.gz 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33042.log 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27415.log 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.33415.log: /var/log/ceph/ceph-client.admin.33042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33415.log.gz 2024-02-03T14:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33042.log.gz 2024-02-03T14:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31475.log 2024-02-03T14:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29029.log 2024-02-03T14:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.27415.log: /var/log/ceph/ceph-client.admin.31475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27415.log.gz 2024-02-03T14:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31475.log.gz 2024-02-03T14:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.29029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29029.log.gz 2024-02-03T14:45:32.682 INFO:teuthology.orchestra.run.smithi007.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-02-03T14:45:32.737 INFO:teuthology.orchestra.run.smithi007.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-02-03T14:45:32.767 INFO:teuthology.orchestra.run.smithi007.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-02-03T14:45:32.778 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-02-03T14:45:33.026 INFO:teuthology.orchestra.run.smithi007.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-02-03T14:45:33.208 INFO:teuthology.orchestra.run.smithi007.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-02-03T14:45:33.209 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-02-03T14:45:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m0.968s 2024-02-03T14:45:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m3.437s 2024-02-03T14:45:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.231s 2024-02-03T14:45:33.211 INFO:tasks.ceph:Archiving logs... 2024-02-03T14:45:33.211 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/remote/smithi007/log 2024-02-03T14:45:33.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-03T14:45:33.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-02-03T14:45:33.889 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-03T14:45:33.899 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-03T14:45:33.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-03T14:45:33.939 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-03T14:45:33.953 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-03T14:45:33.953 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:*hv01.front.sepi 44.4.53.2 2 u 44 64 377 0.119 -1.634 0.594 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:+hv02.front.sepi 63.231.80.2 3 u 49 64 377 0.082 -0.822 0.527 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:+hv03.front.sepi 74.6.168.72 3 u 54 64 377 0.061 -0.779 0.435 2024-02-03T14:45:33.966 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-03T14:45:33.967 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-03T14:45:34.009 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-03T14:45:34.010 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-03T14:45:34.059 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-03T14:45:34.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-03T14:45:34.159 INFO:teuthology.task.internal:Duration was 1005.691924 seconds 2024-02-03T14:45:34.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-03T14:45:34.234 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-03T14:45:34.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-03T14:45:34.275 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-03T14:45:34.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-02-03T14:45:34.275 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-03T14:45:34.329 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-03T14:45:34.329 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-03T14:45:34.396 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-03T14:45:34.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-03T14:45:34.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-03T14:45:34.502 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-03T14:45:34.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-03T14:45:34.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-03T14:45:34.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-03T14:45:34.566 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-02-03T14:45:34.586 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-03T14:45:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:45:34.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-03T14:45:34.641 INFO:teuthology.task.internal:Transferring archived files... 2024-02-03T14:45:34.642 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661/remote/smithi007 2024-02-03T14:45:34.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-03T14:45:34.694 INFO:teuthology.task.internal:Removing archive directory... 2024-02-03T14:45:34.695 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-03T14:45:34.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-03T14:45:34.745 INFO:teuthology.task.internal:Not uploading archives. 2024-02-03T14:45:34.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-03T14:45:34.791 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-03T14:45:34.791 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-03T14:45:34.798 INFO:teuthology.orchestra.run.smithi007.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 3 14:45 /home/ubuntu/cephtest 2024-02-03T14:45:34.798 INFO:teuthology.orchestra.run.smithi007.stdout: 674540 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 3 14:38 /home/ubuntu/cephtest/mnt.0 2024-02-03T14:45:34.799 INFO:teuthology.orchestra.run.smithi007.stdout: 674541 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 3 14:39 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-03T14:45:34.799 INFO:teuthology.orchestra.run.smithi007.stdout: 686450 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 3 14:42 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-03T14:45:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-03T14:45:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-03T14:45:34.800 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-03T14:45:34.807 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-03T14:45:34.844 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-03T14:45:34.901 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi007 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-03T14:45:34.901 INFO:teuthology.nuke:Checking targets against current locks 2024-02-03T14:45:34.928 DEBUG:teuthology.nuke:shortname: smithi007 2024-02-03T14:45:34.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-03T14:45:34.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-03 14:20:27.685066', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBPpZzp9pCH8Oyjw0rR2MIR8xJuJlvu1OfcdxY/MrRHXXCpOCkj6W+EKTcUzxtjaXzj9OXQA1QBr9gnoMODTd8I='} 2024-02-03T14:45:35.002 INFO:teuthology.orchestra.console.smithi007:Power off 2024-02-03T14:45:35.002 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-03T14:45:35.027 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-02-03T14:45:35.028 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-03T14:45:35.050 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-02-03T14:45:39.052 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-03T14:45:39.162 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-02-03T14:45:43.167 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-03T14:45:43.278 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-02-03T14:45:47.281 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-03T14:45:47.395 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-02-03T14:45:47.496 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-02-03T14:45:47.602 INFO:teuthology.run:Summary data: description: rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} duration: 1005.6919240951538 failure_reason: 'Command failed (workunit test cephtool/test.sh) on smithi007 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh''' flavor: default owner: scheduled_pdiazbou@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f49f43b0d8e04abf9cdea9c2dcd60927 status: fail success: false 2024-02-03T14:45:47.602 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-03T14:45:47.700 INFO:teuthology.run:FAIL