2023-11-03T13:38:31.777 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-03T13:38:31.778 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_sseshasa@teuthology --verbose --archive /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381 --description rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} --name sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi -- /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/orig.config.yaml 2023-11-03T13:38:31.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-03T13:38:31.893 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381 branch: wip-sseshasa-testing-2023-11-02-1957 description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} email: null first_in_suite: false job_id: '7445381' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sseshasa-testing-2023-11-02-1957 ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sseshasa-testing-2023-11-02-1957 sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a owner: scheduled_sseshasa@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3972 sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a sleep_before_teardown: 0 subset: 222/150000 suite: rados suite_branch: wip-sseshasa-testing-2023-11-02-1957 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWvSLCWudOvyAAkiWxaMd3EWqD2QOGdrg1rRjM3vPXSLunrPnXTJl/QHegXRGFOvdmGgnOtsiCgWjLBVxOhgcE= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-03_06:54:32 tube: smithi user: sseshasa verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-03T13:38:31.894 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa; will attempt to use it 2023-11-03T13:38:31.895 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks 2023-11-03T13:38:31.895 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-03T13:38:31.897 INFO:teuthology.task.internal:Checking packages... 2023-11-03T13:38:31.921 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '10145169af62d2c41c01dd8795eeddb1b869c21a' 2023-11-03T13:38:31.921 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-03T13:38:31.922 INFO:teuthology.packaging:ref: None 2023-11-03T13:38:31.922 INFO:teuthology.packaging:tag: None 2023-11-03T13:38:31.922 INFO:teuthology.packaging:branch: wip-sseshasa-testing-2023-11-02-1957 2023-11-03T13:38:31.923 INFO:teuthology.packaging:sha1: 10145169af62d2c41c01dd8795eeddb1b869c21a 2023-11-03T13:38:31.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-sseshasa-testing-2023-11-02-1957 2023-11-03T13:38:32.141 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6870.g10145169 2023-11-03T13:38:32.141 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-03T13:38:32.152 INFO:teuthology.task.internal:no buildpackages task found 2023-11-03T13:38:32.152 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-03T13:38:32.166 INFO:teuthology.task.internal:Saving configuration 2023-11-03T13:38:32.182 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-03T13:38:32.189 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-03T13:38:32.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-03 13:33:39.511881', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWvSLCWudOvyAAkiWxaMd3EWqD2QOGdrg1rRjM3vPXSLunrPnXTJl/QHegXRGFOvdmGgnOtsiCgWjLBVxOhgcE='} 2023-11-03T13:38:32.211 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-03T13:38:32.216 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-03T13:38:32.217 INFO:teuthology.run_tasks:Running task console_log... 2023-11-03T13:38:32.264 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0939390a60>, signals=[15]) 2023-11-03T13:38:32.266 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-03T13:38:32.272 INFO:teuthology.task.internal:Opening connections... 2023-11-03T13:38:32.272 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:38:32.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-03T13:38:32.360 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-03T13:38:32.367 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2023-11-03T13:38:32.430 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2023-11-03T13:38:32.431 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2023-11-03T13:38:32.487 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2023-11-03T13:38:32.488 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2023-11-03T13:38:32.488 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2023-11-03T13:38:32.488 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2023-11-03T13:38:32.488 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2023-11-03T13:38:32.488 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2023-11-03T13:38:32.489 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-03T13:38:32.489 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2023-11-03T13:38:32.489 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2023-11-03T13:38:32.489 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-03T13:38:32.489 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2023-11-03T13:38:32.490 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-03T13:38:32.490 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-03T13:38:32.490 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-03T13:38:32.491 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2023-11-03T13:38:32.514 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-03T13:38:32.522 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-03T13:38:32.527 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-03T13:38:32.527 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2023-11-03T13:38:32.543 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-03T13:38:32.548 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-03T13:38:32.549 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2023-11-03T13:38:32.619 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-03T13:38:32.651 INFO:teuthology.run_tasks:Running task kernel... 2023-11-03T13:38:32.665 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-03T13:38:32.666 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-03T13:38:32.667 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-03T13:38:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:38:32.683 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-11-03T13:38:32.741 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-361.el9.x86_64 2023-11-03T13:38:32.742 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-361.el9.x86_64 2023-11-03T13:38:32.742 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2023-11-03T13:38:34.768 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-11-03T13:38:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-11-03T13:38:36.951 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2023-11-03T13:38:39.885 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 18 MB 00:01 2023-11-03T13:38:46.028 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.8 MB 00:02 2023-11-03T13:38:47.645 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2023-11-03T13:38:49.105 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 14 MB/s | 19 MB 00:01 2023-11-03T13:38:53.613 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-11-03T13:38:55.719 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-03T13:38:55.720 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-03T13:38:55.812 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:38:55.817 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:38:55.817 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-11-03T13:38:55.818 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:38:55.818 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:38:55.818 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-11-03T13:38:55.818 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:38:55.819 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-11-03T13:38:55.819 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-11-03T13:38:55.819 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-11-03T13:38:55.819 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:38:55.820 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:38:55.820 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:38:55.820 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2023-11-03T13:38:55.820 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:38:55.821 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 95 M 2023-11-03T13:38:55.821 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 120 M 2023-11-03T13:38:55.821 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:38:56.264 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 14 MB/s | 5.3 MB 00:00 2023-11-03T13:38:56.816 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 22 MB/s | 20 MB 00:00 2023-11-03T13:38:57.440 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 24 MB/s | 38 MB 00:01 2023-11-03T13:38:58.073 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 18 MB/s | 32 MB 00:01 2023-11-03T13:38:58.074 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:38:58.074 INFO:teuthology.orchestra.run.smithi073.stdout:Total 42 MB/s | 95 MB 00:02 2023-11-03T13:38:58.920 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:38:59.282 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:38:59.282 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:39:00.614 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:39:00.615 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:39:03.547 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:39:04.163 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-11-03T13:39:04.281 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-03T13:39:07.022 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-03T13:39:07.217 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-03T13:39:11.851 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-03T13:39:11.980 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:16.639 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:45.756 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:45.798 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:46.245 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:46.245 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-11-03T13:39:46.245 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-03T13:39:46.246 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-03T13:39:46.632 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-03T13:39:46.632 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:39:46.632 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:39:46.633 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-378.el9.x86_64 2023-11-03T13:39:46.633 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-11-03T13:39:46.633 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-11-03T13:39:46.633 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-11-03T13:39:46.633 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:39:46.634 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:39:47.147 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-11-03T13:39:47.303 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-03T13:39:47.303 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-03T13:39:47.304 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-11-03T13:39:47.304 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-03T13:39:47.304 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-03T13:39:47.304 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-03T13:39:47.305 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2023-11-03T13:39:47.305 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-11-03T13:39:47.421 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-03T13:39:47.422 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-03T13:39:47.422 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2023-11-03T13:39:47.467 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2023-11-03T13:39:47.467 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2023-11-03T13:39:47.468 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.06 2023-11-03T13:39:47.468 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 68.el9 2023-11-03T13:39:47.468 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2023-11-03T13:39:47.468 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-03T13:39:47.468 INFO:teuthology.orchestra.run.smithi073.stdout:Group : Unspecified 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 8264254 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-03T13:39:47.469 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-03T13:39:47.470 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : builder@centos.org 2023-11-03T13:39:47.470 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2023-11-03T13:39:47.470 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-03T13:39:47.470 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2023-11-03T13:39:47.471 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2023-11-03T13:39:47.471 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:39:47.472 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-03T13:39:47.472 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-03T13:39:47.472 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-03T13:39:47.472 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2023-11-03T13:39:47.472 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:39:47.473 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2023-11-03T13:39:47.474 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-03T13:39:47.474 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 5.14.0-378.el9.x86_64 2023-11-03T13:39:47.474 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-03T13:39:48.130 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2023-11-03T13:39:49.622 INFO:teuthology.orchestra.run.smithi073.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-03T13:39:49.648 INFO:teuthology.orchestra.run.smithi073.stderr:done 2023-11-03T13:39:49.650 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-11-03T13:39:49.666 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.HzikPgqi1F 2023-11-03T13:39:49.666 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HzikPgqi1F 2023-11-03T13:39:49.736 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.HzikPgqi1F 2023-11-03T13:39:49.878 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.HzikPgqi1F is 6KB 2023-11-03T13:39:49.898 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.HzikPgqi1F 2023-11-03T13:39:49.912 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2023-11-03T13:39:49.980 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-03T13:39:49.981 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-03T13:39:49.981 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-03T13:39:49.981 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-11-03T13:39:49.982 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-11-03T13:39:50.123 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2023-11-03T13:40:20.154 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-03T13:40:20.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:40:20.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-03T13:40:38.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-11-03T13:40:47.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:40:47.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-03T13:40:50.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-11-03T13:41:02.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:41:02.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-03T13:41:34.597 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-11-03T13:41:34.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:41:34.961 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-11-03T13:41:34.962 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-11-03T13:41:34.976 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-378.el9.x86_64 2023-11-03T13:41:34.976 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-11-03T13:41:34.977 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-03T13:41:34.977 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-03T13:41:35.978 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-03T13:41:35.979 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-03T13:41:36.084 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2023-11-03T13:41:36.119 DEBUG:teuthology.parallel:result is None 2023-11-03T13:41:36.119 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-03T13:41:36.130 INFO:teuthology.task.internal:Creating test directory... 2023-11-03T13:41:36.130 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-03T13:41:36.145 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-03T13:41:36.153 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-03T13:41:36.161 INFO:teuthology.task.internal:Creating archive directory... 2023-11-03T13:41:36.161 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-03T13:41:36.217 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-03T13:41:36.224 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-03T13:41:36.225 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-03T13:41:36.284 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-03T13:41:36.296 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-03T13:41:36.298 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-03T13:41:36.306 INFO:teuthology.task.internal:Configuring sudo... 2023-11-03T13:41:36.307 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-03T13:41:36.389 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-03T13:41:36.398 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-03T13:41:36.399 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-03T13:41:36.446 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-03T13:41:36.531 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-03T13:41:36.588 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-03T13:41:36.671 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-03T13:41:36.718 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:41:36.719 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-03T13:41:36.788 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2023-11-03T13:41:36.935 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-03T13:41:37.443 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-03T13:41:37.451 INFO:teuthology.task.internal:Starting timer... 2023-11-03T13:41:37.451 INFO:teuthology.run_tasks:Running task pcp... 2023-11-03T13:41:37.466 INFO:teuthology.run_tasks:Running task selinux... 2023-11-03T13:41:37.477 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2023-11-03T13:41:37.587 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2023-11-03T13:41:37.593 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-03T13:41:37.593 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2023-11-03T13:41:37.618 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2023-11-03T13:41:37.619 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi073.front.sepia.ceph.com': 'permissive'} 2023-11-03T13:41:37.619 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-03T13:41:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:41:37.709 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-03T13:41:37.710 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2023-11-03T13:41:37.829 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-03T13:41:37.840 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-03T13:41:37.841 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-03T13:41:37.858 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-03T13:41:37.859 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-03T13:47:18.418 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com')] 2023-11-03T13:47:18.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:47:18.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-03T13:47:18.506 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-11-03T13:47:18.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-11-03T13:47:18.575 INFO:teuthology.run_tasks:Running task clock... 2023-11-03T13:47:18.586 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-03T13:47:18.587 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-03T13:47:18.588 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-03T13:47:18.645 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-03T13:47:18.661 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-03T13:47:18.690 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2023-11-03T13:47:18.704 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2023-11-03T13:47:18.720 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-03T13:47:18.736 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-03T13:47:18.780 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2023-11-03T13:47:18.783 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-03T13:47:18.783 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-11-03T13:47:18.784 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-03T13:47:18.784 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-03T13:47:18.784 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-03T13:47:18.784 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-03T13:47:18.785 INFO:teuthology.run_tasks:Running task install... 2023-11-03T13:47:18.794 DEBUG:teuthology.task.install:project ceph 2023-11-03T13:47:18.794 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a'}} 2023-11-03T13:47:18.794 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a'} 2023-11-03T13:47:18.795 INFO:teuthology.task.install:Using flavor: default 2023-11-03T13:47:18.801 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-03T13:47:18.802 INFO:teuthology.task.install:extra packages: [] 2023-11-03T13:47:18.802 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a', 'tag': None, 'wait_for_package': False} 2023-11-03T13:47:18.803 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=10145169af62d2c41c01dd8795eeddb1b869c21a 2023-11-03T13:47:19.029 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-11-02-1957/10145169af62d2c41c01dd8795eeddb1b869c21a/centos/9/flavors/default/ 2023-11-03T13:47:19.030 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6870.g10145169 2023-11-03T13:47:19.204 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-11-02-1957/10145169af62d2c41c01dd8795eeddb1b869c21a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-11-02-1957/10145169af62d2c41c01dd8795eeddb1b869c21a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-sseshasa-testing-2023-11-02-1957/10145169af62d2c41c01dd8795eeddb1b869c21a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-03T13:47:19.205 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:47:19.205 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-03T13:47:19.235 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-03T13:47:19.235 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/10145169af62d2c41c01dd8795eeddb1b869c21a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-03T13:47:19.303 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-03T13:47:19.383 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-03T13:47:19.450 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2023-11-03T13:47:19.451 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2023-11-03T13:47:19.699 INFO:teuthology.orchestra.run.smithi073.stdout:72 files removed 2023-11-03T13:47:19.726 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw 2023-11-03T13:47:20.465 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 184 kB/s | 78 kB 00:00 2023-11-03T13:47:20.891 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-11-03T13:47:21.408 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2023-11-03T13:47:21.769 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-11-03T13:47:23.053 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-11-03T13:47:25.177 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 7.9 MB 00:01 2023-11-03T13:47:30.078 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 18 MB 00:03 2023-11-03T13:47:33.871 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-11-03T13:47:36.271 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2023-11-03T13:47:40.293 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 19 MB 00:03 2023-11-03T13:47:44.914 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-11-03T13:47:47.291 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:47:47.294 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:47:47.294 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:47:47.295 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:47:47.295 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:47:47.295 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:18.0.0-6870.g10145169.el9 ceph 8.6 M 2023-11-03T13:47:47.296 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2023-11-03T13:47:47.296 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:18.0.0-6870.g10145169.el9 ceph 3.3 M 2023-11-03T13:47:47.296 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:18.0.0-6870.g10145169.el9 ceph 3.0 M 2023-11-03T13:47:47.296 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:47:47.297 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options 2023-11-03T13:47:47.297 INFO:teuthology.orchestra.run.smithi073.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-03T13:47:47.297 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:18.0.0-6870.g10145169.el9 ceph 5.2 M 2023-11-03T13:47:47.297 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:18.0.0-6870.g10145169.el9 ceph 19 M 2023-11-03T13:47:47.298 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:18.0.0-6870.g10145169.el9 ceph 24 k 2023-11-03T13:47:47.298 INFO:teuthology.orchestra.run.smithi073.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-11-03T13:47:47.298 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-03T13:47:47.299 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-03T13:47:47.299 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-03T13:47:47.299 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:18.0.0-6870.g10145169.el9 ceph 716 k 2023-11-03T13:47:47.299 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-03T13:47:47.299 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-03T13:47:47.300 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-03T13:47:47.300 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:18.0.0-6870.g10145169.el9 ceph 472 k 2023-11-03T13:47:47.300 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-03T13:47:47.300 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:18.0.0-6870.g10145169.el9 ceph 4.8 M 2023-11-03T13:47:47.301 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-03T13:47:47.301 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-03T13:47:47.301 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-03T13:47:47.302 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse 2023-11-03T13:47:47.302 INFO:teuthology.orchestra.run.smithi073.stdout: x86_64 2:18.0.0-6870.g10145169.el9 ceph 45 k 2023-11-03T13:47:47.302 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:18.0.0-6870.g10145169.el9 ceph 127 k 2023-11-03T13:47:47.303 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:18.0.0-6870.g10145169.el9 ceph 161 k 2023-11-03T13:47:47.303 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-03T13:47:47.303 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:18.0.0-6870.g10145169.el9 ceph 321 k 2023-11-03T13:47:47.303 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:18.0.0-6870.g10145169.el9 ceph 298 k 2023-11-03T13:47:47.304 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:18.0.0-6870.g10145169.el9 ceph 99 k 2023-11-03T13:47:47.304 INFO:teuthology.orchestra.run.smithi073.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-03T13:47:47.304 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-03T13:47:47.304 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:47:47.304 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:47:47.305 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:47:47.305 INFO:teuthology.orchestra.run.smithi073.stdout:Install 28 Packages 2023-11-03T13:47:47.305 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2023-11-03T13:47:47.305 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:47:47.306 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 55 M 2023-11-03T13:47:47.306 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:47:48.307 INFO:teuthology.orchestra.run.smithi073.stdout:(1/30): ceph-base-18.0.0-6870.g10145169.el9.x86 7.5 MB/s | 5.2 MB 00:00 2023-11-03T13:47:48.341 INFO:teuthology.orchestra.run.smithi073.stdout:(2/30): ceph-selinux-18.0.0-6870.g10145169.el9. 724 kB/s | 24 kB 00:00 2023-11-03T13:47:48.416 INFO:teuthology.orchestra.run.smithi073.stdout:(3/30): libcephfs2-18.0.0-6870.g10145169.el9.x8 9.3 MB/s | 716 kB 00:00 2023-11-03T13:47:48.474 INFO:teuthology.orchestra.run.smithi073.stdout:(4/30): libradosstriper1-18.0.0-6870.g10145169. 7.9 MB/s | 472 kB 00:00 2023-11-03T13:47:48.733 INFO:teuthology.orchestra.run.smithi073.stdout:(5/30): librgw2-18.0.0-6870.g10145169.el9.x86_6 19 MB/s | 4.8 MB 00:00 2023-11-03T13:47:48.767 INFO:teuthology.orchestra.run.smithi073.stdout:(6/30): python3-ceph-argparse-18.0.0-6870.g1014 1.3 MB/s | 45 kB 00:00 2023-11-03T13:47:48.809 INFO:teuthology.orchestra.run.smithi073.stdout:(7/30): python3-ceph-common-18.0.0-6870.g101451 3.0 MB/s | 127 kB 00:00 2023-11-03T13:47:48.862 INFO:teuthology.orchestra.run.smithi073.stdout:(8/30): python3-cephfs-18.0.0-6870.g10145169.el 3.2 MB/s | 161 kB 00:00 2023-11-03T13:47:48.909 INFO:teuthology.orchestra.run.smithi073.stdout:(9/30): python3-rados-18.0.0-6870.g10145169.el9 6.3 MB/s | 321 kB 00:00 2023-11-03T13:47:48.968 INFO:teuthology.orchestra.run.smithi073.stdout:(10/30): python3-rbd-18.0.0-6870.g10145169.el9. 5.0 MB/s | 298 kB 00:00 2023-11-03T13:47:49.010 INFO:teuthology.orchestra.run.smithi073.stdout:(11/30): python3-rgw-18.0.0-6870.g10145169.el9. 2.3 MB/s | 99 kB 00:00 2023-11-03T13:47:49.102 INFO:teuthology.orchestra.run.smithi073.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.1 MB/s | 106 kB 00:00 2023-11-03T13:47:49.136 INFO:teuthology.orchestra.run.smithi073.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2023-11-03T13:47:49.194 INFO:teuthology.orchestra.run.smithi073.stdout:(14/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-11-03T13:47:49.228 INFO:teuthology.orchestra.run.smithi073.stdout:(15/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-11-03T13:47:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:(16/30): python3-prettytable-0.7.2-27.el9.noarc 986 kB/s | 41 kB 00:00 2023-11-03T13:47:49.695 INFO:teuthology.orchestra.run.smithi073.stdout:(17/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 106 kB/s | 45 kB 00:00 2023-11-03T13:47:49.979 INFO:teuthology.orchestra.run.smithi073.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 391 kB/s | 111 kB 00:00 2023-11-03T13:47:50.396 INFO:teuthology.orchestra.run.smithi073.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 741 kB/s | 309 kB 00:00 2023-11-03T13:47:51.188 INFO:teuthology.orchestra.run.smithi073.stdout:(20/30): ceph-common-18.0.0-6870.g10145169.el9. 5.3 MB/s | 19 MB 00:03 2023-11-03T13:47:51.305 INFO:teuthology.orchestra.run.smithi073.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 209 kB/s | 24 kB 00:00 2023-11-03T13:47:51.388 INFO:teuthology.orchestra.run.smithi073.stdout:(22/30): libarrow-9.0.0-9.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2023-11-03T13:47:51.564 INFO:teuthology.orchestra.run.smithi073.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 188 kB/s | 49 kB 00:00 2023-11-03T13:47:51.589 INFO:teuthology.orchestra.run.smithi073.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2023-11-03T13:47:51.856 INFO:teuthology.orchestra.run.smithi073.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 718 kB/s | 192 kB 00:00 2023-11-03T13:47:51.964 INFO:teuthology.orchestra.run.smithi073.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.0 MB/s | 837 kB 00:00 2023-11-03T13:47:52.248 INFO:teuthology.orchestra.run.smithi073.stdout:(27/30): librados2-18.0.0-6870.g10145169.el9.x8 12 MB/s | 3.3 MB 00:00 2023-11-03T13:47:52.407 INFO:teuthology.orchestra.run.smithi073.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2023-11-03T13:47:52.457 INFO:teuthology.orchestra.run.smithi073.stdout:(29/30): librbd1-18.0.0-6870.g10145169.el9.x86_ 15 MB/s | 3.0 MB 00:00 2023-11-03T13:47:53.415 INFO:teuthology.orchestra.run.smithi073.stdout:(30/30): ceph-radosgw-18.0.0-6870.g10145169.el9 1.5 MB/s | 8.6 MB 00:05 2023-11-03T13:47:53.421 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:47:53.421 INFO:teuthology.orchestra.run.smithi073.stdout:Total 9.0 MB/s | 55 MB 00:06 2023-11-03T13:47:53.432 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:47:53.470 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:47:53.471 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:47:53.937 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:47:53.938 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:47:55.316 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:47:55.378 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-11-03T13:47:55.497 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-11-03T13:47:55.579 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-11-03T13:47:55.893 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-11-03T13:47:55.945 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:18.0.0-6870.g10145169.el9.x86_64 5/32 2023-11-03T13:47:56.040 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:18.0.0-6870.g10145169.el9.x86_64 5/32 2023-11-03T13:47:56.130 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:18.0.0-6870.g10145169.el9.x86_64 6/32 2023-11-03T13:47:56.171 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:18.0.0-6870.g10145169.el9.x86_64 7/32 2023-11-03T13:47:56.234 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:18.0.0-6870.g10145169.el9.x86_64 7/32 2023-11-03T13:47:56.262 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-11-03T13:47:56.306 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:18.0.0-6870.g10145169.el9. 9/32 2023-11-03T13:47:56.372 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:18.0.0-6870.g10145169.el9.x86_64 10/32 2023-11-03T13:47:56.399 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:18.0.0-6870.g10145169.el9.x86_6 11/32 2023-11-03T13:47:56.464 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6870.g10145169.el9.x86_6 11/32 2023-11-03T13:47:56.569 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-11-03T13:47:56.606 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-11-03T13:47:56.652 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-11-03T13:47:56.688 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-11-03T13:47:56.715 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-11-03T13:47:56.750 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-11-03T13:47:56.783 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-03T13:47:57.092 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-11-03T13:47:57.122 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:18.0.0-6870.g10145169.el9.x86_64 20/32 2023-11-03T13:47:57.217 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:18.0.0-6870.g10145169.el9.x86_64 20/32 2023-11-03T13:47:57.246 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:18.0.0-6870.g10145169.el9.x86_64 21/32 2023-11-03T13:47:57.282 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-11-03T13:47:57.708 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-11-03T13:47:57.736 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:18.0.0-6870.g10145169.el9.x86_64 24/32 2023-11-03T13:47:57.807 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:18.0.0-6870.g10145169.el9.x86_64 24/32 2023-11-03T13:47:57.882 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:18.0.0-6870.g10145169.el9.x86_64 25/32 2023-11-03T13:47:57.913 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:18.0.0-6870.g10145169.el9.x8 26/32 2023-11-03T13:48:00.043 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 27/32 2023-11-03T13:48:00.077 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 27/32 2023-11-03T13:48:00.641 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 27/32 2023-11-03T13:48:00.689 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:18.0.0-6870.g10145169.el9.x86_64 28/32 2023-11-03T13:48:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:18.0.0-6870.g10145169.el9.x86_64 28/32 2023-11-03T13:48:00.779 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-03T13:48:00.779 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-03T13:48:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:00.807 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:18.0.0-6870.g10145169.el9.x86_64 29/32 2023-11-03T13:48:10.189 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6870.g10145169.el9.x86_64 29/32 2023-11-03T13:48:10.189 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2023-11-03T13:48:10.189 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2023-11-03T13:48:10.190 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2023-11-03T13:48:10.190 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2023-11-03T13:48:10.190 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2023-11-03T13:48:10.190 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /root 2023-11-03T13:48:10.190 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2023-11-03T13:48:10.191 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:10.977 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:18.0.0-6870.g10145169.el9.x86_64 30/32 2023-11-03T13:48:11.001 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6870.g10145169.el9.x86_64 30/32 2023-11-03T13:48:11.001 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:11.001 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-03T13:48:11.001 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-03T13:48:11.002 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-03T13:48:11.002 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:11.002 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-03T13:48:11.079 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-03T13:48:11.079 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-03T13:48:12.193 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-03T13:48:12.193 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:18.0.0-6870.g10145169.el9.x86_64 1/32 2023-11-03T13:48:12.193 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 2/32 2023-11-03T13:48:12.194 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:18.0.0-6870.g10145169.el9.x86_64 3/32 2023-11-03T13:48:12.194 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:18.0.0-6870.g10145169.el9.x86_64 4/32 2023-11-03T13:48:12.194 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:18.0.0-6870.g10145169.el9.x86_64 5/32 2023-11-03T13:48:12.194 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:18.0.0-6870.g10145169.el9.x86_6 6/32 2023-11-03T13:48:12.194 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:18.0.0-6870.g10145169.el9.x86_64 7/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6870.g10145169.el9. 8/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:18.0.0-6870.g10145169.el9.x8 9/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:18.0.0-6870.g10145169.el9.x86_64 10/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:18.0.0-6870.g10145169.el9.x86_64 11/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:18.0.0-6870.g10145169.el9.x86_64 12/32 2023-11-03T13:48:12.195 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:18.0.0-6870.g10145169.el9.x86_64 13/32 2023-11-03T13:48:12.196 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-11-03T13:48:12.196 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-11-03T13:48:12.196 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-11-03T13:48:12.196 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-11-03T13:48:12.196 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-03T13:48:12.197 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-11-03T13:48:12.197 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-11-03T13:48:12.197 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-11-03T13:48:12.197 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-11-03T13:48:12.197 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-11-03T13:48:12.198 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-11-03T13:48:12.198 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-11-03T13:48:12.198 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-11-03T13:48:12.198 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-11-03T13:48:12.198 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-11-03T13:48:12.199 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:18.0.0-6870.g10145169.el9.x86_64 29/32 2023-11-03T13:48:12.199 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-11-03T13:48:12.199 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:18.0.0-6870.g10145169.el9.x86_64 31/32 2023-11-03T13:48:13.180 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-11-03T13:48:13.180 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:13.181 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2023-11-03T13:48:13.181 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.181 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.181 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:13.182 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-03T13:48:13.182 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.182 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.182 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.182 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.183 INFO:teuthology.orchestra.run.smithi073.stdout: fmt-8.1.1-5.el9.x86_64 2023-11-03T13:48:13.183 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-03T13:48:13.183 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-03T13:48:13.183 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-03T13:48:13.183 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.184 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-03T13:48:13.184 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-03T13:48:13.184 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-03T13:48:13.184 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.184 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-03T13:48:13.185 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.185 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-03T13:48:13.185 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-03T13:48:13.185 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-03T13:48:13.185 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.186 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.186 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.186 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-03T13:48:13.186 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.186 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.187 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:13.187 INFO:teuthology.orchestra.run.smithi073.stdout: re2-1:20211101-3.el9.x86_64 2023-11-03T13:48:13.187 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-03T13:48:13.187 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:13.187 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:13.543 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-test 2023-11-03T13:48:14.081 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:30 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:14.223 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:14.223 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:14.223 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:48:14.224 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:14.224 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:14.224 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:18.0.0-6870.g10145169.el9 ceph 41 M 2023-11-03T13:48:14.224 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:48:14.224 INFO:teuthology.orchestra.run.smithi073.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-11-03T13:48:14.225 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:18.0.0-6870.g10145169.el9 ceph 164 k 2023-11-03T13:48:14.225 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-03T13:48:14.225 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-03T13:48:14.225 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-03T13:48:14.226 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:14.226 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:14.226 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:14.226 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2023-11-03T13:48:14.226 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:14.227 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 42 M 2023-11-03T13:48:14.228 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 172 M 2023-11-03T13:48:14.228 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:14.436 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2023-11-03T13:48:14.477 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2023-11-03T13:48:14.511 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-03T13:48:14.562 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): libcephsqlite-18.0.0-6870.g10145169.el9. 621 kB/s | 164 kB 00:00 2023-11-03T13:48:14.679 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-11-03T13:48:17.655 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): ceph-test-18.0.0-6870.g10145169.el9.x86_ 12 MB/s | 41 MB 00:03 2023-11-03T13:48:17.657 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:17.657 INFO:teuthology.orchestra.run.smithi073.stdout:Total 12 MB/s | 42 MB 00:03 2023-11-03T13:48:17.675 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:17.692 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:17.693 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:18.050 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:18.050 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:18.614 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:18.679 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-03T13:48:18.713 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-03T13:48:18.744 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-03T13:48:18.788 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-11-03T13:48:18.815 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:18.0.0-6870.g10145169.el9.x86_64 5/6 2023-11-03T13:48:22.678 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6870.g10145169.el9.x86_64 5/6 2023-11-03T13:48:22.747 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:18.0.0-6870.g10145169.el9.x86_64 6/6 2023-11-03T13:48:24.960 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-test-2:18.0.0-6870.g10145169.el9.x86_64 6/6 2023-11-03T13:48:24.960 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:18.0.0-6870.g10145169.el9.x86_64 1/6 2023-11-03T13:48:24.960 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:18.0.0-6870.g10145169.el9.x86_64 2/6 2023-11-03T13:48:24.961 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-03T13:48:24.961 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-11-03T13:48:24.961 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-11-03T13:48:25.362 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-11-03T13:48:25.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:25.363 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:25.363 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:25.363 INFO:teuthology.orchestra.run.smithi073.stdout: jq-1.6-15.el9.x86_64 2023-11-03T13:48:25.364 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:25.364 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-03T13:48:25.364 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-03T13:48:25.364 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-03T13:48:25.364 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:25.365 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:25.675 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph 2023-11-03T13:48:26.217 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:42 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:26.371 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:26.373 INFO:teuthology.orchestra.run.smithi073.stdout:========================================================================================= 2023-11-03T13:48:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:48:26.373 INFO:teuthology.orchestra.run.smithi073.stdout:========================================================================================= 2023-11-03T13:48:26.374 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:18.0.0-6870.g10145169.el9 ceph 6.5 k 2023-11-03T13:48:26.374 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:48:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:18.0.0-6870.g10145169.el9 ceph 2.1 M 2023-11-03T13:48:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:18.0.0-6870.g10145169.el9 ceph 1.5 M 2023-11-03T13:48:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 243 k 2023-11-03T13:48:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:18.0.0-6870.g10145169.el9 ceph 4.4 M 2023-11-03T13:48:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:18.0.0-6870.g10145169.el9 ceph 18 M 2023-11-03T13:48:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-03T13:48:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-03T13:48:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-03T13:48:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-03T13:48:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-03T13:48:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-03T13:48:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-03T13:48:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-03T13:48:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-03T13:48:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-03T13:48:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-03T13:48:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-03T13:48:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-03T13:48:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-03T13:48:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-03T13:48:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-03T13:48:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-03T13:48:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-03T13:48:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-03T13:48:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-03T13:48:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-03T13:48:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-03T13:48:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:26.381 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:26.381 INFO:teuthology.orchestra.run.smithi073.stdout:========================================================================================= 2023-11-03T13:48:26.381 INFO:teuthology.orchestra.run.smithi073.stdout:Install 33 Packages 2023-11-03T13:48:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:26.382 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 30 M 2023-11-03T13:48:26.382 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 110 M 2023-11-03T13:48:26.382 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:26.959 INFO:teuthology.orchestra.run.smithi073.stdout:(1/33): ceph-18.0.0-6870.g10145169.el9.x86_64.r 19 kB/s | 6.5 kB 00:00 2023-11-03T13:48:27.659 INFO:teuthology.orchestra.run.smithi073.stdout:(2/33): ceph-mgr-18.0.0-6870.g10145169.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-11-03T13:48:28.017 INFO:teuthology.orchestra.run.smithi073.stdout:(3/33): ceph-mds-18.0.0-6870.g10145169.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2023-11-03T13:48:28.143 INFO:teuthology.orchestra.run.smithi073.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6870.g1014 1.9 MB/s | 243 kB 00:00 2023-11-03T13:48:28.243 INFO:teuthology.orchestra.run.smithi073.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-03T13:48:28.268 INFO:teuthology.orchestra.run.smithi073.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-11-03T13:48:28.293 INFO:teuthology.orchestra.run.smithi073.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-11-03T13:48:28.335 INFO:teuthology.orchestra.run.smithi073.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-11-03T13:48:28.369 INFO:teuthology.orchestra.run.smithi073.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-03T13:48:28.403 INFO:teuthology.orchestra.run.smithi073.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-03T13:48:28.436 INFO:teuthology.orchestra.run.smithi073.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-03T13:48:28.470 INFO:teuthology.orchestra.run.smithi073.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-11-03T13:48:28.678 INFO:teuthology.orchestra.run.smithi073.stdout:(13/33): ceph-mon-18.0.0-6870.g10145169.el9.x86 2.5 MB/s | 4.4 MB 00:01 2023-11-03T13:48:28.795 INFO:teuthology.orchestra.run.smithi073.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 3.8 MB/s | 1.2 MB 00:00 2023-11-03T13:48:28.954 INFO:teuthology.orchestra.run.smithi073.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 275 kB/s | 43 kB 00:00 2023-11-03T13:48:29.054 INFO:teuthology.orchestra.run.smithi073.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2023-11-03T13:48:29.079 INFO:teuthology.orchestra.run.smithi073.stdout:(17/33): python3-pytz-2021.1-5.el9.noarch.rpm 127 kB/s | 51 kB 00:00 2023-11-03T13:48:29.155 INFO:teuthology.orchestra.run.smithi073.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2023-11-03T13:48:29.180 INFO:teuthology.orchestra.run.smithi073.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-11-03T13:48:29.205 INFO:teuthology.orchestra.run.smithi073.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2023-11-03T13:48:29.231 INFO:teuthology.orchestra.run.smithi073.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2023-11-03T13:48:29.264 INFO:teuthology.orchestra.run.smithi073.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2023-11-03T13:48:29.290 INFO:teuthology.orchestra.run.smithi073.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2023-11-03T13:48:29.323 INFO:teuthology.orchestra.run.smithi073.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 792 kB/s | 46 kB 00:00 2023-11-03T13:48:29.374 INFO:teuthology.orchestra.run.smithi073.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2023-11-03T13:48:29.399 INFO:teuthology.orchestra.run.smithi073.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2023-11-03T13:48:29.433 INFO:teuthology.orchestra.run.smithi073.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2023-11-03T13:48:29.458 INFO:teuthology.orchestra.run.smithi073.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-11-03T13:48:29.491 INFO:teuthology.orchestra.run.smithi073.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2023-11-03T13:48:29.517 INFO:teuthology.orchestra.run.smithi073.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-11-03T13:48:29.575 INFO:teuthology.orchestra.run.smithi073.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2023-11-03T13:48:29.609 INFO:teuthology.orchestra.run.smithi073.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2023-11-03T13:48:30.609 INFO:teuthology.orchestra.run.smithi073.stdout:(33/33): ceph-osd-18.0.0-6870.g10145169.el9.x86 6.0 MB/s | 18 MB 00:02 2023-11-03T13:48:30.613 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:30.613 INFO:teuthology.orchestra.run.smithi073.stdout:Total 7.1 MB/s | 30 MB 00:04 2023-11-03T13:48:30.647 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:30.688 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:30.688 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:31.051 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:31.052 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:32.290 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:32.346 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-03T13:48:32.385 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-03T13:48:32.410 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-03T13:48:32.501 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-03T13:48:32.580 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-03T13:48:32.620 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-03T13:48:32.659 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-03T13:48:32.726 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-03T13:48:32.780 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-03T13:48:32.819 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-03T13:48:32.868 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-03T13:48:32.929 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-03T13:48:33.115 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-03T13:48:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-03T13:48:33.204 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-03T13:48:33.238 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-03T13:48:33.268 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-03T13:48:33.343 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-03T13:48:33.380 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-03T13:48:33.413 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-03T13:48:33.455 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-03T13:48:33.514 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-03T13:48:33.602 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-03T13:48:33.636 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-03T13:48:33.709 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-03T13:48:33.844 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-03T13:48:33.969 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-03T13:48:34.129 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6870.g10145169.el9. 28/33 2023-11-03T13:48:34.159 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:18.0.0-6870.g10145169.el9.x86_64 29/33 2023-11-03T13:48:34.183 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6870.g10145169.el9.x86_64 29/33 2023-11-03T13:48:34.184 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:34.184 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-03T13:48:34.184 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-03T13:48:34.184 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-03T13:48:34.185 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:35.760 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:18.0.0-6870.g10145169.el9.x86_64 30/33 2023-11-03T13:48:35.787 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:18.0.0-6870.g10145169.el9.x86_64 30/33 2023-11-03T13:48:35.788 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:35.788 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-03T13:48:35.788 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-03T13:48:35.788 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-03T13:48:35.789 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:36.229 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:18.0.0-6870.g10145169.el9.x86_64 31/33 2023-11-03T13:48:36.250 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:18.0.0-6870.g10145169.el9.x86_64 31/33 2023-11-03T13:48:36.250 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:36.251 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-03T13:48:36.251 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-03T13:48:36.251 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-03T13:48:36.251 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:36.461 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:18.0.0-6870.g10145169.el9.x86_64 32/33 2023-11-03T13:48:36.484 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:18.0.0-6870.g10145169.el9.x86_64 32/33 2023-11-03T13:48:36.484 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:36.484 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-03T13:48:36.484 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-03T13:48:36.485 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-03T13:48:36.485 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:36.550 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:18.0.0-6870.g10145169.el9.x86_64 33/33 2023-11-03T13:48:38.336 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-2:18.0.0-6870.g10145169.el9.x86_64 33/33 2023-11-03T13:48:38.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:18.0.0-6870.g10145169.el9.x86_64 1/33 2023-11-03T13:48:38.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:18.0.0-6870.g10145169.el9.x86_64 2/33 2023-11-03T13:48:38.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:18.0.0-6870.g10145169.el9.x86_64 3/33 2023-11-03T13:48:38.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:18.0.0-6870.g10145169.el9.x86_64 4/33 2023-11-03T13:48:38.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:18.0.0-6870.g10145169.el9.x86_64 5/33 2023-11-03T13:48:38.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6870.g10145169.el9. 6/33 2023-11-03T13:48:38.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-03T13:48:38.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-03T13:48:38.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-03T13:48:38.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-03T13:48:38.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-03T13:48:38.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-03T13:48:38.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-03T13:48:38.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-03T13:48:38.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-03T13:48:38.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-03T13:48:38.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-03T13:48:38.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-03T13:48:38.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-03T13:48:38.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-03T13:48:38.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-03T13:48:38.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-03T13:48:38.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-03T13:48:38.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-03T13:48:38.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-03T13:48:38.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-03T13:48:38.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-03T13:48:38.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-03T13:48:38.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-03T13:48:38.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-03T13:48:38.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-03T13:48:38.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-03T13:48:39.234 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-03T13:48:39.235 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:39.235 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:39.235 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:39.236 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-03T13:48:39.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-03T13:48:39.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-03T13:48:39.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-03T13:48:39.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-03T13:48:39.237 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-03T13:48:39.238 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-03T13:48:39.238 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-03T13:48:39.238 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-03T13:48:39.238 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-03T13:48:39.238 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-03T13:48:39.239 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-03T13:48:39.239 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-03T13:48:39.239 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-03T13:48:39.239 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-03T13:48:39.239 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply-3.11-14.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-03T13:48:39.240 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-03T13:48:39.241 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-03T13:48:39.241 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-03T13:48:39.241 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-03T13:48:39.241 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-03T13:48:39.241 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-03T13:48:39.242 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:39.242 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:39.566 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-base 2023-11-03T13:48:40.109 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:56 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:40.200 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-base-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:48:40.249 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:40.250 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:48:40.250 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:40.307 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install cephadm 2023-11-03T13:48:40.844 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:56 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:40.982 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:40.983 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:40.984 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:48:40.984 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:40.984 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:40.984 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 270 k 2023-11-03T13:48:40.984 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:40.985 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:40.985 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:40.985 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:48:40.985 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:40.985 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 270 k 2023-11-03T13:48:40.986 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 265 k 2023-11-03T13:48:40.986 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:41.210 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm-18.0.0-6870.g10145169.el9.noarch.rpm 1.2 MB/s | 270 kB 00:00 2023-11-03T13:48:41.211 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:41.211 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.2 MB/s | 270 kB 00:00 2023-11-03T13:48:41.211 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:41.216 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:41.216 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:41.219 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:41.219 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:41.288 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:41.766 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:48:41.848 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:48:42.201 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:48:42.534 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:48:42.535 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:42.535 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:42.535 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:42.535 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:42.536 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:42.723 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-immutable-object-cache 2023-11-03T13:48:43.275 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:59 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:43.419 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:43.420 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:43.420 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2023-11-03T13:48:43.420 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:43.420 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:43.421 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6870.g10145169.el9 ceph 142 k 2023-11-03T13:48:43.421 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:43.421 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:43.421 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:43.421 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:48:43.422 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:43.422 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 142 k 2023-11-03T13:48:43.422 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 400 k 2023-11-03T13:48:43.422 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-immutable-object-cache-18.0.0-6870.g101451 694 kB/s | 142 kB 00:00 2023-11-03T13:48:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:Total 689 kB/s | 142 kB 00:00 2023-11-03T13:48:43.626 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:43.632 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:43.633 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:43.684 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:43.684 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:43.820 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:43.884 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6870.g10145169. 1/1 2023-11-03T13:48:43.907 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6870.g10145169. 1/1 2023-11-03T13:48:43.908 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:48:43.908 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-03T13:48:43.908 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:44.875 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6870.g10145169. 1/1 2023-11-03T13:48:44.875 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:44.875 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:44.875 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:48:44.876 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:44.876 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:45.131 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr 2023-11-03T13:48:45.681 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:01 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:45.781 INFO:teuthology.orchestra.run.smithi073.stdout:Package ceph-mgr-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:48:45.831 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:45.831 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:48:45.832 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:45.886 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-dashboard 2023-11-03T13:48:46.425 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:02 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:46.566 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:46.566 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:46.567 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:48:46.567 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:46.567 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:46.567 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 1.8 M 2023-11-03T13:48:46.567 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:48:46.568 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 23 k 2023-11-03T13:48:46.568 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 15 k 2023-11-03T13:48:46.568 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-03T13:48:46.568 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-03T13:48:46.568 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-03T13:48:46.569 INFO:teuthology.orchestra.run.smithi073.stdout:Installing weak dependencies: 2023-11-03T13:48:46.569 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-03T13:48:46.569 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:46.569 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:46.569 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:48:46.570 INFO:teuthology.orchestra.run.smithi073.stdout:Install 7 Packages 2023-11-03T13:48:46.570 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:46.570 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.1 M 2023-11-03T13:48:46.571 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 8.6 M 2023-11-03T13:48:46.571 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:46.911 INFO:teuthology.orchestra.run.smithi073.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6870.g101 93 kB/s | 23 kB 00:00 2023-11-03T13:48:47.012 INFO:teuthology.orchestra.run.smithi073.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6870.g1014 43 kB/s | 15 kB 00:00 2023-11-03T13:48:47.070 INFO:teuthology.orchestra.run.smithi073.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2023-11-03T13:48:47.187 INFO:teuthology.orchestra.run.smithi073.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 327 kB/s | 31 kB 00:00 2023-11-03T13:48:47.271 INFO:teuthology.orchestra.run.smithi073.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 159 kB/s | 41 kB 00:00 2023-11-03T13:48:47.313 INFO:teuthology.orchestra.run.smithi073.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2023-11-03T13:48:47.621 INFO:teuthology.orchestra.run.smithi073.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6870.g10145169 1.8 MB/s | 1.8 MB 00:00 2023-11-03T13:48:47.624 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:47.625 INFO:teuthology.orchestra.run.smithi073.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-11-03T13:48:47.625 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:47.632 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:47.633 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:47.698 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:47.699 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:48.000 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:48.064 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-03T13:48:48.097 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-03T13:48:48.152 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-03T13:48:48.184 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-03T13:48:48.213 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6870.g10145169.el9.n 5/7 2023-11-03T13:48:48.462 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6870.g10145169.el9. 6/7 2023-11-03T13:48:48.500 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6870.g10145169.el9.noarc 7/7 2023-11-03T13:48:49.033 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6870.g10145169.el9.noarc 7/7 2023-11-03T13:48:49.033 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6870.g10145169.el9. 1/7 2023-11-03T13:48:49.033 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6870.g10145169.el9.noarc 2/7 2023-11-03T13:48:49.033 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6870.g10145169.el9.n 3/7 2023-11-03T13:48:49.034 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-03T13:48:49.034 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-03T13:48:49.034 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-03T13:48:49.368 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-03T13:48:49.368 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:49.368 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:49.369 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:49.369 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:49.369 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:49.369 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-03T13:48:49.369 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-03T13:48:49.370 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-03T13:48:49.370 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-03T13:48:49.370 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:49.370 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:48:49.571 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-03T13:48:50.109 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:06 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:48:50.249 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:48:50.250 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2023-11-03T13:48:50.250 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:48:50.250 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2023-11-03T13:48:50.251 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:48:50.251 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 7.4 M 2023-11-03T13:48:50.251 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:48:50.251 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-03T13:48:50.251 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-03T13:48:50.252 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-03T13:48:50.252 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-03T13:48:50.252 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-03T13:48:50.252 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-03T13:48:50.253 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:50.254 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:48:50.254 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2023-11-03T13:48:50.254 INFO:teuthology.orchestra.run.smithi073.stdout:Install 12 Packages 2023-11-03T13:48:50.254 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:50.255 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 37 M 2023-11-03T13:48:50.255 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 209 M 2023-11-03T13:48:50.255 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:48:50.378 INFO:teuthology.orchestra.run.smithi073.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 253 kB/s | 32 kB 00:00 2023-11-03T13:48:50.403 INFO:teuthology.orchestra.run.smithi073.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 712 kB/s | 17 kB 00:00 2023-11-03T13:48:50.428 INFO:teuthology.orchestra.run.smithi073.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2023-11-03T13:48:50.487 INFO:teuthology.orchestra.run.smithi073.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-11-03T13:48:50.612 INFO:teuthology.orchestra.run.smithi073.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-11-03T13:48:50.637 INFO:teuthology.orchestra.run.smithi073.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 205 kB 00:00 2023-11-03T13:48:50.713 INFO:teuthology.orchestra.run.smithi073.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-11-03T13:48:50.738 INFO:teuthology.orchestra.run.smithi073.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-03T13:48:50.947 INFO:teuthology.orchestra.run.smithi073.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2023-11-03T13:48:50.972 INFO:teuthology.orchestra.run.smithi073.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2023-11-03T13:48:50.997 INFO:teuthology.orchestra.run.smithi073.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-11-03T13:48:52.031 INFO:teuthology.orchestra.run.smithi073.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.2 MB/s | 7.4 MB 00:01 2023-11-03T13:48:52.033 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:48:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:Total 21 MB/s | 37 MB 00:01 2023-11-03T13:48:52.034 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:48:52.057 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:48:52.057 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:48:52.474 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:48:52.474 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:48:53.181 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:48:53.253 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-03T13:48:53.285 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-03T13:48:53.379 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-03T13:48:53.413 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-03T13:48:53.581 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-03T13:48:53.604 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-03T13:48:53.700 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-03T13:48:54.380 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-03T13:48:54.490 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-03T13:48:55.728 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-03T13:48:56.746 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-03T13:48:56.800 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6870.g10145 12/12 2023-11-03T13:48:59.255 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6870.g10145 12/12 2023-11-03T13:48:59.256 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6870.g10145 1/12 2023-11-03T13:48:59.256 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-03T13:48:59.256 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-03T13:48:59.256 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-03T13:48:59.257 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-03T13:48:59.257 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-03T13:48:59.257 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-03T13:48:59.257 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-03T13:48:59.257 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-03T13:48:59.258 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-03T13:48:59.258 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-03T13:48:59.719 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-03T13:48:59.719 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:59.719 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:48:59.719 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:48:59.720 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-03T13:48:59.720 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-03T13:48:59.720 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-03T13:48:59.720 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-03T13:48:59.720 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-03T13:48:59.721 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-03T13:48:59.721 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-03T13:48:59.721 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-03T13:48:59.721 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-03T13:48:59.721 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-03T13:48:59.722 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-03T13:48:59.722 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:48:59.722 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:00.080 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-rook 2023-11-03T13:49:00.633 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:16 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:00.775 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:00.778 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:00.778 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:49:00.778 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:00.778 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 48 k 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-03T13:49:00.779 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-03T13:49:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-03T13:49:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-03T13:49:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-03T13:49:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-03T13:49:00.780 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-03T13:49:00.781 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-03T13:49:00.781 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-03T13:49:00.781 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-03T13:49:00.781 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:00.781 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:00.782 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:00.782 INFO:teuthology.orchestra.run.smithi073.stdout:Install 13 Packages 2023-11-03T13:49:00.782 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:00.782 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 2.0 M 2023-11-03T13:49:00.783 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 26 M 2023-11-03T13:49:00.783 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:01.053 INFO:teuthology.orchestra.run.smithi073.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 211 kB/s | 24 kB 00:00 2023-11-03T13:49:01.078 INFO:teuthology.orchestra.run.smithi073.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2023-11-03T13:49:01.120 INFO:teuthology.orchestra.run.smithi073.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-11-03T13:49:01.145 INFO:teuthology.orchestra.run.smithi073.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-11-03T13:49:01.171 INFO:teuthology.orchestra.run.smithi073.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-03T13:49:01.196 INFO:teuthology.orchestra.run.smithi073.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-11-03T13:49:01.263 INFO:teuthology.orchestra.run.smithi073.stdout:(7/13): ceph-mgr-rook-18.0.0-6870.g10145169.el9 146 kB/s | 48 kB 00:00 2023-11-03T13:49:01.588 INFO:teuthology.orchestra.run.smithi073.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2023-11-03T13:49:01.655 INFO:teuthology.orchestra.run.smithi073.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 66 kB/s | 32 kB 00:00 2023-11-03T13:49:01.872 INFO:teuthology.orchestra.run.smithi073.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2023-11-03T13:49:01.972 INFO:teuthology.orchestra.run.smithi073.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 278 kB/s | 197 kB 00:00 2023-11-03T13:49:02.039 INFO:teuthology.orchestra.run.smithi073.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2023-11-03T13:49:02.248 INFO:teuthology.orchestra.run.smithi073.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-11-03T13:49:02.252 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:02.252 INFO:teuthology.orchestra.run.smithi073.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-11-03T13:49:02.253 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:02.266 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:02.266 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:02.365 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:02.373 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:02.887 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:03.052 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-03T13:49:03.115 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-03T13:49:03.170 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-03T13:49:03.200 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-03T13:49:03.241 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-03T13:49:03.333 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-03T13:49:03.434 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-03T13:49:03.484 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-03T13:49:04.203 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-03T13:49:04.248 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-03T13:49:04.282 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-03T13:49:04.328 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-03T13:49:04.352 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:18.0.0-6870.g10145169.el9.noarch 13/13 2023-11-03T13:49:05.162 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6870.g10145169.el9.noarch 13/13 2023-11-03T13:49:05.163 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6870.g10145169.el9.noarch 1/13 2023-11-03T13:49:05.163 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-03T13:49:05.163 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-03T13:49:05.163 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-03T13:49:05.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-03T13:49:05.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-03T13:49:05.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-03T13:49:05.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-03T13:49:05.164 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-03T13:49:05.165 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-03T13:49:05.165 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-03T13:49:05.165 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-03T13:49:05.569 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-03T13:49:05.569 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:05.569 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:05.569 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:49:05.570 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-03T13:49:05.570 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-03T13:49:05.570 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-03T13:49:05.570 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-03T13:49:05.570 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-03T13:49:05.571 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-03T13:49:05.571 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-03T13:49:05.571 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-03T13:49:05.571 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-03T13:49:05.571 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-03T13:49:05.572 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-03T13:49:05.572 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-03T13:49:05.572 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:05.572 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:05.796 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-mgr-cephadm 2023-11-03T13:49:06.345 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:22 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:06.487 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:06.488 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:06.488 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:49:06.488 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:06.488 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:06.488 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 132 k 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-03T13:49:06.489 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-03T13:49:06.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:06.490 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:06.490 INFO:teuthology.orchestra.run.smithi073.stdout:======================================================================================== 2023-11-03T13:49:06.490 INFO:teuthology.orchestra.run.smithi073.stdout:Install 6 Packages 2023-11-03T13:49:06.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:06.491 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 6.8 M 2023-11-03T13:49:06.491 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 33 M 2023-11-03T13:49:06.492 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:06.734 INFO:teuthology.orchestra.run.smithi073.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-11-03T13:49:06.901 INFO:teuthology.orchestra.run.smithi073.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-11-03T13:49:07.059 INFO:teuthology.orchestra.run.smithi073.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2023-11-03T13:49:07.076 INFO:teuthology.orchestra.run.smithi073.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2023-11-03T13:49:07.093 INFO:teuthology.orchestra.run.smithi073.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2023-11-03T13:49:07.176 INFO:teuthology.orchestra.run.smithi073.stdout:(6/6): ceph-mgr-cephadm-18.0.0-6870.g10145169.e 216 kB/s | 132 kB 00:00 2023-11-03T13:49:07.180 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:07.180 INFO:teuthology.orchestra.run.smithi073.stdout:Total 9.8 MB/s | 6.8 MB 00:00 2023-11-03T13:49:07.180 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:07.189 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:07.190 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:07.278 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:07.278 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:07.572 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:07.750 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-03T13:49:07.811 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-03T13:49:08.324 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-03T13:49:08.426 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-03T13:49:08.491 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-03T13:49:08.533 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6870.g10145169.el9.noarch 6/6 2023-11-03T13:49:09.285 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6870.g10145169.el9.noarch 6/6 2023-11-03T13:49:09.285 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6870.g10145169.el9.noarch 1/6 2023-11-03T13:49:09.285 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-03T13:49:09.285 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-03T13:49:09.286 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-03T13:49:09.286 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-03T13:49:09.604 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-03T13:49:09.604 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:09.604 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:09.604 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-03T13:49:09.605 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:09.606 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:09.849 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-fuse 2023-11-03T13:49:10.394 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:26 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:10.533 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:10.534 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:10.535 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:49:10.535 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:10.535 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:10.535 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:18.0.0-6870.g10145169.el9 ceph 855 k 2023-11-03T13:49:10.536 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-11-03T13:49:10.536 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-03T13:49:10.536 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:10.537 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:10.537 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:10.537 INFO:teuthology.orchestra.run.smithi073.stdout:Install 2 Packages 2023-11-03T13:49:10.537 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:10.537 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 934 k 2023-11-03T13:49:10.538 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 2.7 M 2023-11-03T13:49:10.538 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:10.645 INFO:teuthology.orchestra.run.smithi073.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 718 kB/s | 79 kB 00:00 2023-11-03T13:49:10.837 INFO:teuthology.orchestra.run.smithi073.stdout:(2/2): ceph-fuse-18.0.0-6870.g10145169.el9.x86_ 2.8 MB/s | 855 kB 00:00 2023-11-03T13:49:10.838 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:10.838 INFO:teuthology.orchestra.run.smithi073.stdout:Total 3.0 MB/s | 934 kB 00:00 2023-11-03T13:49:10.839 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:10.858 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:10.859 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:10.933 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:10.934 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:11.113 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:11.243 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-03T13:49:11.336 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:18.0.0-6870.g10145169.el9.x86_64 2/2 2023-11-03T13:49:11.998 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6870.g10145169.el9.x86_64 2/2 2023-11-03T13:49:11.998 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:18.0.0-6870.g10145169.el9.x86_64 1/2 2023-11-03T13:49:12.354 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-03T13:49:12.354 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:12.354 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:12.355 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:18.0.0-6870.g10145169.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-03T13:49:12.355 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:12.355 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:12.656 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-volume 2023-11-03T13:49:13.215 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:29 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:13.358 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:13.359 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:13.359 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-11-03T13:49:13.359 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:13.360 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:13.360 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume noarch 2:18.0.0-6870.g10145169.el9 ceph-noarch 262 k 2023-11-03T13:49:13.360 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:13.360 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:13.360 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:13.361 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:13.361 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:13.361 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 262 k 2023-11-03T13:49:13.361 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 1.2 M 2023-11-03T13:49:13.361 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:13.741 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-volume-18.0.0-6870.g10145169.el9.noarch.rp 688 kB/s | 262 kB 00:00 2023-11-03T13:49:13.742 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:13.742 INFO:teuthology.orchestra.run.smithi073.stdout:Total 685 kB/s | 262 kB 00:00 2023-11-03T13:49:13.742 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:13.750 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:13.750 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:13.786 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:13.787 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:14.004 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:14.076 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-volume-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:49:14.091 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-volume-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:49:14.092 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:49:14.092 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-03T13:49:14.092 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:15.126 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-volume-2:18.0.0-6870.g10145169.el9.noarch 1/1 2023-11-03T13:49:15.127 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:15.127 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:15.127 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume-2:18.0.0-6870.g10145169.el9.noarch 2023-11-03T13:49:15.127 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:15.128 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:15.409 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados-devel 2023-11-03T13:49:15.979 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:31 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:16.125 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:16.126 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:16.127 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2023-11-03T13:49:16.127 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:16.127 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:16.127 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:18.0.0-6870.g10145169.el9 ceph 126 k 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:16.128 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 126 k 2023-11-03T13:49:16.129 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 456 k 2023-11-03T13:49:16.129 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:16.338 INFO:teuthology.orchestra.run.smithi073.stdout:librados-devel-18.0.0-6870.g10145169.el9.x86_64 601 kB/s | 126 kB 00:00 2023-11-03T13:49:16.339 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:16.339 INFO:teuthology.orchestra.run.smithi073.stdout:Total 597 kB/s | 126 kB 00:00 2023-11-03T13:49:16.339 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:16.344 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:16.344 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:16.399 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:16.400 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:16.549 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:16.653 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:17.098 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:17.429 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:17.430 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:17.430 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:17.430 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:49:17.430 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:17.431 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:17.663 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs2 2023-11-03T13:49:18.220 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:34 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:18.313 INFO:teuthology.orchestra.run.smithi073.stdout:Package libcephfs2-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:18.363 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:18.364 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:18.364 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:18.426 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install libcephfs-devel 2023-11-03T13:49:18.973 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:34 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:19.115 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:19.116 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:19.117 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2023-11-03T13:49:19.117 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:19.117 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:19.117 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:18.0.0-6870.g10145169.el9 ceph 31 k 2023-11-03T13:49:19.117 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:19.118 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:19.118 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:19.118 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:19.118 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:19.118 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 31 k 2023-11-03T13:49:19.119 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 139 k 2023-11-03T13:49:19.119 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:19.452 INFO:teuthology.orchestra.run.smithi073.stdout:libcephfs-devel-18.0.0-6870.g10145169.el9.x86_6 93 kB/s | 31 kB 00:00 2023-11-03T13:49:19.453 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:19.453 INFO:teuthology.orchestra.run.smithi073.stdout:Total 93 kB/s | 31 kB 00:00 2023-11-03T13:49:19.453 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:19.456 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:19.457 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:19.481 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:19.482 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:19.582 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:19.700 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:20.131 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:20.388 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:20.389 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:20.389 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:20.389 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:49:20.389 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:20.390 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:20.553 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librados2 2023-11-03T13:49:21.099 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:37 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:21.189 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:21.238 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:21.239 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:21.240 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:21.309 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install librbd1 2023-11-03T13:49:21.855 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:37 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:21.946 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:21.996 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:21.997 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:21.997 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:22.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rados 2023-11-03T13:49:22.599 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:38 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:22.691 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rados-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:22.740 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:22.741 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:22.741 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:22.805 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rgw 2023-11-03T13:49:23.344 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:39 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:23.435 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rgw-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:23.484 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:23.485 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:23.486 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:23.555 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-cephfs 2023-11-03T13:49:24.100 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:40 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:24.193 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-cephfs-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:24.243 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:24.244 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:24.244 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:24.317 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install python3-rbd 2023-11-03T13:49:24.862 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:40 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:24.953 INFO:teuthology.orchestra.run.smithi073.stdout:Package python3-rbd-2:18.0.0-6870.g10145169.el9.x86_64 is already installed. 2023-11-03T13:49:25.003 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:25.004 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-11-03T13:49:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:25.066 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-fuse 2023-11-03T13:49:25.610 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:41 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:25.749 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:25.750 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:25.750 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-11-03T13:49:25.750 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:18.0.0-6870.g10145169.el9 ceph 85 k 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:25.751 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:25.752 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:25.752 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 85 k 2023-11-03T13:49:25.752 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 227 k 2023-11-03T13:49:25.752 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:25.963 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-fuse-18.0.0-6870.g10145169.el9.x86_64.rpm 401 kB/s | 85 kB 00:00 2023-11-03T13:49:25.964 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:25.964 INFO:teuthology.orchestra.run.smithi073.stdout:Total 398 kB/s | 85 kB 00:00 2023-11-03T13:49:25.964 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:25.971 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:25.971 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:26.026 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:26.026 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:26.953 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:49:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:26.953 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:27.183 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-mirror 2023-11-03T13:49:27.734 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:43 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:27.877 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:27.878 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repo Size 2023-11-03T13:49:27.879 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:27.879 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:18.0.0-6870.g10145169.el9 ceph 3.0 M 2023-11-03T13:49:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:27.879 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:27.880 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:27.880 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:27.880 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 3.0 M 2023-11-03T13:49:27.880 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 12 M 2023-11-03T13:49:27.881 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:28.408 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-mirror-18.0.0-6870.g10145169.el9.x86_64.rpm 5.7 MB/s | 3.0 MB 00:00 2023-11-03T13:49:28.409 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:28.409 INFO:teuthology.orchestra.run.smithi073.stdout:Total 5.7 MB/s | 3.0 MB 00:00 2023-11-03T13:49:28.409 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:28.418 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:28.418 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:28.489 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:28.490 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:28.898 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:28.967 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:28.991 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:28.992 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-03T13:49:28.992 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-03T13:49:28.992 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-03T13:49:28.992 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-03T13:49:28.993 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:30.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:30.096 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:30.096 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:30.097 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:49:30.097 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:30.097 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:30.327 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install rbd-nbd 2023-11-03T13:49:30.885 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:46 ago on Fri 03 Nov 2023 01:47:44 PM UTC. 2023-11-03T13:49:31.025 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-11-03T13:49:31.026 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:31.026 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2023-11-03T13:49:31.026 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:31.026 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-11-03T13:49:31.027 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:18.0.0-6870.g10145169.el9 ceph 169 k 2023-11-03T13:49:31.027 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:31.027 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-11-03T13:49:31.027 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2023-11-03T13:49:31.027 INFO:teuthology.orchestra.run.smithi073.stdout:Install 1 Package 2023-11-03T13:49:31.028 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:31.028 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 169 k 2023-11-03T13:49:31.028 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 490 k 2023-11-03T13:49:31.028 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-11-03T13:49:31.378 INFO:teuthology.orchestra.run.smithi073.stdout:rbd-nbd-18.0.0-6870.g10145169.el9.x86_64.rpm 481 kB/s | 169 kB 00:00 2023-11-03T13:49:31.379 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-11-03T13:49:31.380 INFO:teuthology.orchestra.run.smithi073.stdout:Total 479 kB/s | 169 kB 00:00 2023-11-03T13:49:31.380 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-11-03T13:49:31.387 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-11-03T13:49:31.387 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-11-03T13:49:31.443 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-11-03T13:49:31.443 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-11-03T13:49:31.619 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-11-03T13:49:31.726 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:32.139 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:32.434 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:18.0.0-6870.g10145169.el9.x86_64 1/1 2023-11-03T13:49:32.434 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:32.435 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-11-03T13:49:32.435 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:18.0.0-6870.g10145169.el9.x86_64 2023-11-03T13:49:32.435 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:32.435 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-11-03T13:49:32.636 DEBUG:teuthology.parallel:result is None 2023-11-03T13:49:32.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=10145169af62d2c41c01dd8795eeddb1b869c21a 2023-11-03T13:49:32.784 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-03T13:49:32.805 INFO:teuthology.orchestra.run.smithi073.stdout:18.0.0-6870.g10145169.el9 2023-11-03T13:49:32.806 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6870.g10145169.el9 2023-11-03T13:49:32.806 INFO:teuthology.task.install:The correct ceph version 18.0.0-6870.g10145169 is installed. 2023-11-03T13:49:32.807 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-03T13:49:32.808 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:32.808 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-03T13:49:32.873 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-03T13:49:32.874 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:32.874 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2023-11-03T13:49:32.940 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-03T13:49:33.006 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-03T13:49:33.006 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:33.007 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-03T13:49:33.073 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-03T13:49:33.140 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-03T13:49:33.140 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:33.141 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2023-11-03T13:49:33.207 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-03T13:49:33.274 INFO:teuthology.run_tasks:Running task ceph... 2023-11-03T13:49:33.354 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-03T13:49:33.355 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /var/log/ceph 2023-11-03T13:49:33.381 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-03T13:49:33.382 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-03T13:49:33.448 INFO:tasks.ceph:Creating extra log directories... 2023-11-03T13:49:33.448 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-03T13:49:33.522 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-03T13:49:33.523 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-03T13:49:33.523 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381', 'branch': 'wip-sseshasa-testing-2023-11-02-1957', 'description': 'rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}', 'email': None, 'first_in_suite': False, 'job_id': '7445381', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sseshasa-testing-2023-11-02-1957'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sseshasa-testing-2023-11-02-1957', 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a'}}, 'owner': 'scheduled_sseshasa@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 3972, 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a', 'sleep_before_teardown': 0, 'subset': '222/150000', 'suite': 'rados', 'suite_branch': 'wip-sseshasa-testing-2023-11-02-1957', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a', 'targets': {'smithi073.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWvSLCWudOvyAAkiWxaMd3EWqD2QOGdrg1rRjM3vPXSLunrPnXTJl/QHegXRGFOvdmGgnOtsiCgWjLBVxOhgcE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '10145169af62d2c41c01dd8795eeddb1b869c21a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool', 'mon/pool_ops.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-03_06:54:32', 'tube': 'smithi', 'user': 'sseshasa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-03T13:49:33.524 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-03T13:49:33.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/run/ceph 2023-11-03T13:49:33.635 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:33.636 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2023-11-03T13:49:33.691 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-03T13:49:33.691 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2023-11-03T13:49:33.745 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-03T13:49:33.746 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-03T13:49:33.746 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-11-03T13:49:33.746 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-03T13:49:33.746 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-11-03T13:49:33.747 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-11-03 13:49:29.224950611 +0000 2023-11-03T13:49:33.747 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-11-03 13:46:54.051448328 +0000 2023-11-03T13:49:33.747 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-11-03 13:46:54.051448328 +0000 2023-11-03T13:49:33.747 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2023-11-03 13:46:54.051448328 +0000 2023-11-03T13:49:33.748 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-03T13:49:33.813 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-11-03T13:49:33.813 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-11-03T13:49:33.814 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000200761 s, 2.6 MB/s 2023-11-03T13:49:33.816 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-03T13:49:33.871 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2023-11-03T13:49:33.926 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-03T13:49:33.927 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-03T13:49:33.927 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-11-03T13:49:33.927 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-03T13:49:33.927 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-11-03T13:49:33.928 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-11-03 13:49:29.224950611 +0000 2023-11-03T13:49:33.928 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-11-03 13:46:54.549436467 +0000 2023-11-03T13:49:33.928 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-11-03 13:46:54.549436467 +0000 2023-11-03T13:49:33.928 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2023-11-03 13:46:54.549436467 +0000 2023-11-03T13:49:33.928 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-03T13:49:33.992 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-11-03T13:49:33.992 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-11-03T13:49:33.992 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000229445 s, 2.2 MB/s 2023-11-03T13:49:33.993 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-03T13:49:34.049 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2023-11-03T13:49:34.104 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-03T13:49:34.105 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-03T13:49:34.105 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-11-03T13:49:34.105 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-03T13:49:34.106 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-11-03T13:49:34.106 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-11-03 13:49:29.224950611 +0000 2023-11-03T13:49:34.108 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-11-03 13:46:55.047424605 +0000 2023-11-03T13:49:34.108 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-11-03 13:46:55.047424605 +0000 2023-11-03T13:49:34.109 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2023-11-03 13:46:55.047424605 +0000 2023-11-03T13:49:34.110 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-03T13:49:34.170 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-11-03T13:49:34.171 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-11-03T13:49:34.171 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000162397 s, 3.2 MB/s 2023-11-03T13:49:34.172 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-03T13:49:34.228 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2023-11-03T13:49:34.282 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-03T13:49:34.283 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-03T13:49:34.283 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-11-03T13:49:34.283 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-03T13:49:34.283 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-11-03T13:49:34.283 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-11-03 13:49:29.225950587 +0000 2023-11-03T13:49:34.284 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-11-03 13:46:55.547412696 +0000 2023-11-03T13:49:34.284 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-11-03 13:46:55.547412696 +0000 2023-11-03T13:49:34.284 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2023-11-03 13:46:55.547412696 +0000 2023-11-03T13:49:34.284 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-03T13:49:34.348 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-11-03T13:49:34.348 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-11-03T13:49:34.348 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000267636 s, 1.9 MB/s 2023-11-03T13:49:34.349 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-03T13:49:34.405 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-03T13:49:34.405 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-03T13:49:34.406 INFO:tasks.ceph:Generating config... 2023-11-03T13:49:34.407 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-03T13:49:34.408 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2023-11-03T13:49:34.408 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-11-03T13:49:34.408 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2023-11-03T13:49:34.408 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2023-11-03T13:49:34.409 INFO:tasks.ceph:[global] ms type = async 2023-11-03T13:49:34.409 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-03T13:49:34.409 INFO:tasks.ceph:[mgr] debug monc = 10 2023-11-03T13:49:34.409 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-03T13:49:34.410 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-03T13:49:34.410 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-03T13:49:34.410 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-03T13:49:34.410 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-03T13:49:34.411 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-03T13:49:34.411 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-03T13:49:34.411 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-03T13:49:34.411 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-03T13:49:34.412 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-11-03T13:49:34.412 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-03T13:49:34.412 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-03T13:49:34.412 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-03T13:49:34.413 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-03T13:49:34.413 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-03T13:49:34.413 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-03T13:49:34.413 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-03T13:49:34.414 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-03T13:49:34.414 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-03T13:49:34.414 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-03T13:49:34.415 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-03T13:49:34.415 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-11-03T13:49:34.415 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-11-03T13:49:34.415 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-03T13:49:34.416 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-03T13:49:34.416 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-03T13:49:34.416 INFO:tasks.ceph:Setting up mon.a... 2023-11-03T13:49:34.417 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-03T13:49:34.489 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.keyring 2023-11-03T13:49:34.492 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-03T13:49:34.537 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-03T13:49:34.603 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.73'), ('mon.b', '[v2:172.21.15.73:3301,v1:172.21.15.73:6790]'), ('mon.c', '[v2:172.21.15.73:3302,v1:172.21.15.73:6791]')] 2023-11-03T13:49:34.604 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.73,[v2:172.21.15.73:3301,v1:172.21.15.73:6790],[v2:172.21.15.73:3302,v1:172.21.15.73:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-11-03T13:49:34.605 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:34.605 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-03T13:49:34.658 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.73 --addv b '[v2:172.21.15.73:3301,v1:172.21.15.73:6790]' --addv c '[v2:172.21.15.73:3302,v1:172.21.15.73:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:34.743 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:34.743 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: generated fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:49:34.743 INFO:teuthology.orchestra.run.smithi073.stdout:setting min_mon_release = pacific 2023-11-03T13:49:34.744 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 0 2023-11-03T13:49:34.744 INFO:teuthology.orchestra.run.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:49:34.744 INFO:teuthology.orchestra.run.smithi073.stdout:last_changed 2023-11-03T13:49:34.742002+0000 2023-11-03T13:49:34.744 INFO:teuthology.orchestra.run.smithi073.stdout:created 2023-11-03T13:49:34.742002+0000 2023-11-03T13:49:34.745 INFO:teuthology.orchestra.run.smithi073.stdout:min_mon_release 16 (pacific) 2023-11-03T13:49:34.745 INFO:teuthology.orchestra.run.smithi073.stdout:election_strategy: 3 2023-11-03T13:49:34.745 INFO:teuthology.orchestra.run.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2023-11-03T13:49:34.745 INFO:teuthology.orchestra.run.smithi073.stdout:1: [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] mon.b 2023-11-03T13:49:34.745 INFO:teuthology.orchestra.run.smithi073.stdout:2: [v2:172.21.15.73:3302/0,v1:172.21.15.73:6791/0] mon.c 2023-11-03T13:49:34.746 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-03T13:49:34.747 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-03T13:49:34.804 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3628883d-b18e-4d59-aee7-61c0fc38535d... 2023-11-03T13:49:34.805 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-03T13:49:34.899 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2023-11-03T13:49:34.900 INFO:teuthology.orchestra.run.smithi073.stdout: chdir = "" 2023-11-03T13:49:34.900 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-03T13:49:34.900 INFO:teuthology.orchestra.run.smithi073.stdout: auth supported = cephx 2023-11-03T13:49:34.900 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.900 INFO:teuthology.orchestra.run.smithi073.stdout: filestore xattr use omap = true 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: mon clock drift allowed = 1.000 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: osd crush chooseleaf type = 0 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: auth debug = true 2023-11-03T13:49:34.901 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.902 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on old message = true 2023-11-03T13:49:34.902 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on bug = true 2023-11-03T13:49:34.902 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.902 INFO:teuthology.orchestra.run.smithi073.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-03T13:49:34.902 INFO:teuthology.orchestra.run.smithi073.stdout: mon pg warn max object skew = 0 2023-11-03T13:49:34.903 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.903 INFO:teuthology.orchestra.run.smithi073.stdout: # disable pg_autoscaler by default for new pools 2023-11-03T13:49:34.903 INFO:teuthology.orchestra.run.smithi073.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-03T13:49:34.903 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.903 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default size = 2 2023-11-03T13:49:34.904 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.904 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow primary affinity = true 2023-11-03T13:49:34.904 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow pg remap = true 2023-11-03T13:49:34.904 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on legacy crush tunables = false 2023-11-03T13:49:34.904 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on crush straw calc version zero = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on no sortbitwise = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on osd down out interval zero = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on too few osds = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-03T13:49:34.905 INFO:teuthology.orchestra.run.smithi073.stdout: mon_allow_pool_size_one = true 2023-11-03T13:49:34.906 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.906 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-03T13:49:34.906 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.906 INFO:teuthology.orchestra.run.smithi073.stdout: osd default data pool replay window = 5 2023-11-03T13:49:34.906 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.907 INFO:teuthology.orchestra.run.smithi073.stdout: mon allow pool delete = true 2023-11-03T13:49:34.907 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.907 INFO:teuthology.orchestra.run.smithi073.stdout: mon cluster log file level = debug 2023-11-03T13:49:34.907 INFO:teuthology.orchestra.run.smithi073.stdout: debug asserts on shutdown = true 2023-11-03T13:49:34.907 INFO:teuthology.orchestra.run.smithi073.stdout: mon health detail to clog = false 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: mon host = "172.21.15.73,[v2:172.21.15.73:3301,v1:172.21.15.73:6790],[v2:172.21.15.73:3302,v1:172.21.15.73:6791]" 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: mon client directed command retry = 5 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: mon client hunt interval max multiple = 2 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: mon election default strategy = 3 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr beacon grace = 90 2023-11-03T13:49:34.908 INFO:teuthology.orchestra.run.smithi073.stdout: ms inject socket failures = 1000 2023-11-03T13:49:34.909 INFO:teuthology.orchestra.run.smithi073.stdout: ms type = async 2023-11-03T13:49:34.909 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:49:34.909 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.909 INFO:teuthology.orchestra.run.smithi073.stdout:[osd] 2023-11-03T13:49:34.909 INFO:teuthology.orchestra.run.smithi073.stdout: osd journal size = 100 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub load threshold = 5.0 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub max interval = 600 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock profile = high_recovery_ops 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.910 INFO:teuthology.orchestra.run.smithi073.stdout: osd recover clone overlap = true 2023-11-03T13:49:34.911 INFO:teuthology.orchestra.run.smithi073.stdout: osd recovery max chunk = 1048576 2023-11-03T13:49:34.911 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.911 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug shutdown = true 2023-11-03T13:49:34.911 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug op order = true 2023-11-03T13:49:34.911 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify stray on activate = true 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: osd open classes on start = true 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug pg log writeout = true 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: osd deep scrub update digest min age = 30 2023-11-03T13:49:34.912 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.913 INFO:teuthology.orchestra.run.smithi073.stdout: osd map max advance = 10 2023-11-03T13:49:34.913 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.913 INFO:teuthology.orchestra.run.smithi073.stdout: journal zero on create = true 2023-11-03T13:49:34.913 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.913 INFO:teuthology.orchestra.run.smithi073.stdout: filestore ondisk finisher threads = 3 2023-11-03T13:49:34.914 INFO:teuthology.orchestra.run.smithi073.stdout: filestore apply finisher threads = 3 2023-11-03T13:49:34.914 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.914 INFO:teuthology.orchestra.run.smithi073.stdout: bdev debug aio = true 2023-11-03T13:49:34.914 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug misdirected ops = true 2023-11-03T13:49:34.914 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore block size = 96636764160 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore compression algorithm = snappy 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore compression mode = aggressive 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore fsck on mount = True 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore zero block detection = True 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluefs = 20 2023-11-03T13:49:34.915 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluestore = 20 2023-11-03T13:49:34.916 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-11-03T13:49:34.916 INFO:teuthology.orchestra.run.smithi073.stdout: debug osd = 20 2023-11-03T13:49:34.916 INFO:teuthology.orchestra.run.smithi073.stdout: debug rocksdb = 10 2023-11-03T13:49:34.916 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd backfillfull_ratio = 0.85 2023-11-03T13:49:34.916 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd full ratio = 0.9 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd nearfull ratio = 0.8 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify cached snaps = True 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify missing on start = True 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: osd failsafe full ratio = 0.95 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock override recovery settings = True 2023-11-03T13:49:34.917 INFO:teuthology.orchestra.run.smithi073.stdout: osd objectstore = bluestore 2023-11-03T13:49:34.918 INFO:teuthology.orchestra.run.smithi073.stdout: osd op queue = debug_random 2023-11-03T13:49:34.918 INFO:teuthology.orchestra.run.smithi073.stdout: osd op queue cut off = debug_random 2023-11-03T13:49:34.918 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.918 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr] 2023-11-03T13:49:34.918 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-11-03T13:49:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: debug mgr = 20 2023-11-03T13:49:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2023-11-03T13:49:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2023-11-03T13:49:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2023-11-03T13:49:34.919 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout: mgr/telemetry/nag = false 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout: debug monc = 10 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout:[mon] 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-11-03T13:49:34.920 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2023-11-03T13:49:34.921 INFO:teuthology.orchestra.run.smithi073.stdout: debug paxos = 20 2023-11-03T13:49:34.921 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2023-11-03T13:49:34.921 INFO:teuthology.orchestra.run.smithi073.stdout: mon data avail warn = 5 2023-11-03T13:49:34.921 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr mkfs grace = 240 2023-11-03T13:49:34.921 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd reporter subtree level = osd 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd prime pg temp = true 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-03T13:49:34.922 INFO:teuthology.orchestra.run.smithi073.stdout: auth mon ticket ttl = 660 # 11m 2023-11-03T13:49:34.923 INFO:teuthology.orchestra.run.smithi073.stdout: auth service ticket ttl = 240 # 4m 2023-11-03T13:49:34.923 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.923 INFO:teuthology.orchestra.run.smithi073.stdout: # don't complain about insecure global_id in the test suite 2023-11-03T13:49:34.923 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-03T13:49:34.924 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-03T13:49:34.924 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.924 INFO:teuthology.orchestra.run.smithi073.stdout: # 1m isn't quite enough 2023-11-03T13:49:34.924 INFO:teuthology.orchestra.run.smithi073.stdout: mon_down_mkfs_grace = 2m 2023-11-03T13:49:34.924 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_filestore_osds = false 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout: mon scrub interval = 300 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout:[client] 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout: rgw cache enabled = true 2023-11-03T13:49:34.925 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable ops log = true 2023-11-03T13:49:34.926 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable usage log = true 2023-11-03T13:49:34.926 INFO:teuthology.orchestra.run.smithi073.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-03T13:49:34.926 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-03T13:49:34.926 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.a] 2023-11-03T13:49:34.926 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.b] 2023-11-03T13:49:34.927 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.c] 2023-11-03T13:49:34.928 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-03T13:49:34.928 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-03T13:49:35.024 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-03T13:49:35.025 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:35.025 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-03T13:49:35.040 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:35.040 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-03T13:49:35.098 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:49:35.099 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:35.099 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-03T13:49:35.099 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-03T13:49:35.177 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:35.178 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:35.242 INFO:tasks.ceph:Setting up mon nodes... 2023-11-03T13:49:35.243 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-03T13:49:35.243 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-03T13:49:35.339 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-03T13:49:35.342 INFO:tasks.ceph:Setting up mds nodes... 2023-11-03T13:49:35.343 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-03T13:49:35.343 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-03T13:49:35.390 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-03T13:49:35.402 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-03T13:49:35.402 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-03T13:49:35.403 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-03T13:49:35.470 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-03T13:49:35.470 INFO:tasks.ceph:role: osd.0 2023-11-03T13:49:35.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:49:35.471 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-03T13:49:35.543 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-03T13:49:35.544 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-03T13:49:35.544 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-03T13:49:35.544 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-03T13:49:35.544 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-03T13:49:35.545 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-11-03T13:49:35.545 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-03T13:49:35.545 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-03T13:49:35.545 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-03T13:49:35.545 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-03T13:49:36.052 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-11-03T13:49:36.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-11-03T13:49:36.054 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-03T13:49:36.159 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-03T13:49:36.195 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-03T13:49:36.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-03T13:49:36.263 INFO:tasks.ceph:role: osd.1 2023-11-03T13:49:36.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:49:36.264 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-03T13:49:36.338 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-03T13:49:36.339 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-03T13:49:36.339 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-03T13:49:36.339 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-03T13:49:36.339 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-03T13:49:36.339 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-11-03T13:49:36.340 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-03T13:49:36.340 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-03T13:49:36.340 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-03T13:49:36.340 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-03T13:49:36.815 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-11-03T13:49:36.817 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-11-03T13:49:36.817 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-03T13:49:36.909 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-03T13:49:36.941 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-03T13:49:37.007 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-03T13:49:37.008 INFO:tasks.ceph:role: osd.2 2023-11-03T13:49:37.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:49:37.008 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-03T13:49:37.082 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-03T13:49:37.082 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-03T13:49:37.083 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-03T13:49:37.084 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-03T13:49:37.084 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-03T13:49:37.619 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-11-03T13:49:37.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-11-03T13:49:37.622 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-03T13:49:37.779 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-03T13:49:37.811 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:37.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.917+0000 7fd489d6e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-03T13:49:37.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.917+0000 7fd489d6e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-03T13:49:37.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.917+0000 7fd489d6e500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-03T13:49:37.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.917+0000 7fd489d6e500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-03T13:49:37.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.917+0000 7fd489d6e500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-03T13:49:37.927 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:37.925+0000 7fd489d6e500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-03T13:49:40.784 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-03T13:49:40.815 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:40.910 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.906+0000 7f86a04b4500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-03T13:49:40.910 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.907+0000 7f86a04b4500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-03T13:49:40.910 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.907+0000 7f86a04b4500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-03T13:49:40.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.907+0000 7f86a04b4500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-03T13:49:40.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.907+0000 7f86a04b4500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-03T13:49:40.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:40.907+0000 7f86a04b4500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-03T13:49:43.756 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-03T13:49:43.788 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:43.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.878+0000 7f51e94da500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-03T13:49:43.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.879+0000 7f51e94da500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-03T13:49:43.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.879+0000 7f51e94da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-03T13:49:43.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.879+0000 7f51e94da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-03T13:49:43.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.879+0000 7f51e94da500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-03T13:49:43.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-11-03T13:49:43.879+0000 7f51e94da500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-03T13:49:46.725 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-03T13:49:46.752 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-03T13:49:46.753 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:46.753 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-03T13:49:46.820 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:46.820 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-03T13:49:46.886 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:46.886 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-03T13:49:46.952 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:46.953 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-03T13:49:47.019 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:49:47.019 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-03T13:49:47.074 INFO:tasks.ceph:Adding keys to all mons... 2023-11-03T13:49:47.075 DEBUG:teuthology.orchestra.run.smithi073:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-03T13:49:47.139 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr.x] 2023-11-03T13:49:47.140 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBv+kRlA/YcFBAAKyb88Vdgn8Y6q8KCBIj6yg== 2023-11-03T13:49:47.140 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.0] 2023-11-03T13:49:47.140 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBx+kRliD+7NhAALla2c0B9ZilflYQD000YvQ== 2023-11-03T13:49:47.140 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.1] 2023-11-03T13:49:47.140 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQB0+kRl4fgaNhAAYb2wZhXJgedkl61JANuB5A== 2023-11-03T13:49:47.141 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.2] 2023-11-03T13:49:47.141 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQB3+kRlAaFxNBAA1Q9vokF9aohFGtJuqPoTYg== 2023-11-03T13:49:47.141 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2023-11-03T13:49:47.141 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBv+kRlHD3+FhAA9yh6noUV5w+C3zvkSMhTOw== 2023-11-03T13:49:47.143 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-03T13:49:47.234 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-03T13:49:47.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-03T13:49:47.337 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-03T13:49:47.389 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-03T13:49:47.442 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-03T13:49:47.443 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-03T13:49:47.469 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-03T13:49:47.566 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-03T13:49:47.593 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-03T13:49:47.661 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-03T13:49:47.758 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-03T13:49:47.784 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-03T13:49:47.851 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-03T13:49:47.949 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-03T13:49:47.976 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-03T13:49:48.030 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-03T13:49:48.031 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-03T13:49:48.031 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-03T13:49:48.072 INFO:tasks.ceph.mon.a:Started 2023-11-03T13:49:48.072 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-03T13:49:48.073 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-03T13:49:48.075 INFO:tasks.ceph.mon.b:Started 2023-11-03T13:49:48.076 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-03T13:49:48.076 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-03T13:49:48.078 INFO:tasks.ceph.mon.c:Started 2023-11-03T13:49:48.078 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-03T13:49:48.079 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-03T13:49:48.079 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-03T13:49:48.081 INFO:tasks.ceph.mgr.x:Started 2023-11-03T13:49:48.082 INFO:tasks.ceph:Setting crush tunables to default 2023-11-03T13:49:48.083 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd crush tunables default 2023-11-03T13:49:59.947 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:49:59.945+0000 7fb08a038040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-03T13:49:59.996 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:49:59.994+0000 7fb08a038040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-03T13:50:00.278 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2023-11-03T13:50:00.290 INFO:tasks.ceph:check_enable_crimson: False 2023-11-03T13:50:00.291 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-03T13:50:00.291 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:50:00.291 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-03T13:50:00.319 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:50:00.319 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-03T13:50:00.387 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:50:00.387 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-03T13:50:00.455 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 47768af5-4f35-4087-bb4b-e0a73a13a74b 0 2023-11-03T13:50:00.535 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.533+0000 7fb08a038040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-03T13:50:00.589 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.587+0000 7fb08a038040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-03T13:50:00.644 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.634+0000 7fb08a038040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-03T13:50:00.677 INFO:teuthology.orchestra.run.smithi073.stdout:0 2023-11-03T13:50:00.688 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 8fbd18a9-0046-46d9-aae9-f460f3751db3 1 2023-11-03T13:50:00.985 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.690+0000 7fb08a038040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-03T13:50:00.985 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.732+0000 7fb08a038040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-03T13:50:00.986 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.831+0000 7fb08a038040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-03T13:50:00.986 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.899+0000 7fb08a038040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-03T13:50:00.986 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.943+0000 7fb08a038040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-03T13:50:00.998 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:00.995+0000 7fb08a038040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-03T13:50:01.046 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.044+0000 7fb08a038040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-03T13:50:01.171 INFO:teuthology.orchestra.run.smithi073.stdout:1 2023-11-03T13:50:01.182 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 8e528929-d22b-4938-8bdc-ce337fa128ff 2 2023-11-03T13:50:01.405 INFO:teuthology.orchestra.run.smithi073.stdout:2 2023-11-03T13:50:01.416 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-03T13:50:01.417 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-03T13:50:01.419 INFO:tasks.ceph.osd.0:Started 2023-11-03T13:50:01.419 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-03T13:50:01.420 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-03T13:50:01.421 INFO:tasks.ceph.osd.1:Started 2023-11-03T13:50:01.422 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-03T13:50:01.422 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-03T13:50:01.424 INFO:tasks.ceph.osd.2:Started 2023-11-03T13:50:01.425 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-03T13:50:01.450 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.449+0000 7fb08a038040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-03T13:50:01.601 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:01.602 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":5,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:50:01.400625+0000","last_up_change":"0.000000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:50:01.612 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-03T13:50:01.613 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-03T13:50:01.613 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.611+0000 7fb08a038040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-03T13:50:01.724 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.722+0000 7fb08a038040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-03T13:50:01.769 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.768+0000 7fb08a038040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-03T13:50:01.867 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.865+0000 7fb08a038040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-03T13:50:01.921 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-03T13:50:01.964 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:01.962+0000 7fb08a038040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-03T13:50:02.021 INFO:tasks.ceph.osd.0.smithi073.stderr:2023-11-03T13:50:02.019+0000 7fb80fd9f500 -1 Falling back to public interface 2023-11-03T13:50:02.024 INFO:tasks.ceph.osd.1.smithi073.stderr:2023-11-03T13:50:02.022+0000 7f5e950b4500 -1 Falling back to public interface 2023-11-03T13:50:02.044 INFO:tasks.ceph.osd.2.smithi073.stderr:2023-11-03T13:50:02.042+0000 7fd9d5394500 -1 Falling back to public interface 2023-11-03T13:50:02.085 INFO:teuthology.misc.health.smithi073.stdout: 2023-11-03T13:50:02.085 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":5,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:50:01.400625+0000","last_up_change":"0.000000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:50:02.094 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-03T13:50:02.111 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:02.109+0000 7fb08a038040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-03T13:50:02.153 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:02.151+0000 7fb08a038040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-03T13:50:02.292 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:02.290+0000 7fb08a038040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-03T13:50:03.094 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:03.092+0000 7fb08a038040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-03T13:50:03.364 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:03.362+0000 7fb08a038040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-03T13:50:03.417 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:03.415+0000 7fb08a038040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-03T13:50:03.519 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:03.517+0000 7fb08a038040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-03T13:50:04.542 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:04.540+0000 7fb02ab3d640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-03T13:50:04.543 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:50:04.540+0000 7fb02ab3d640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-03T13:50:05.473 INFO:tasks.ceph.osd.1.smithi073.stderr:2023-11-03T13:50:05.471+0000 7f5e950b4500 -1 osd.1 0 log_to_monitors true 2023-11-03T13:50:05.516 INFO:tasks.ceph.osd.0.smithi073.stderr:2023-11-03T13:50:05.514+0000 7fb80fd9f500 -1 osd.0 0 log_to_monitors true 2023-11-03T13:50:05.561 INFO:tasks.ceph.osd.2.smithi073.stderr:2023-11-03T13:50:05.559+0000 7fd9d5394500 -1 osd.2 0 log_to_monitors true 2023-11-03T13:50:06.548 INFO:tasks.ceph.osd.0.smithi073.stderr:2023-11-03T13:50:06.545+0000 7fb80bd2d640 -1 osd.0 0 waiting for initial osdmap 2023-11-03T13:50:06.552 INFO:tasks.ceph.osd.1.smithi073.stderr:2023-11-03T13:50:06.551+0000 7f5e91040640 -1 osd.1 0 waiting for initial osdmap 2023-11-03T13:50:06.556 INFO:tasks.ceph.osd.1.smithi073.stderr:2023-11-03T13:50:06.554+0000 7f5e8ce68640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-03T13:50:06.567 INFO:tasks.ceph.osd.0.smithi073.stderr:2023-11-03T13:50:06.565+0000 7fb807b55640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-03T13:50:08.143 INFO:tasks.ceph.osd.2.smithi073.stderr:2023-11-03T13:50:08.141+0000 7fd9d1b38640 -1 osd.2 0 waiting for initial osdmap 2023-11-03T13:50:08.153 INFO:tasks.ceph.osd.2.smithi073.stderr:2023-11-03T13:50:08.151+0000 7fd9c9142640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-03T13:50:08.397 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-03T13:50:08.673 INFO:teuthology.misc.health.smithi073.stdout: 2023-11-03T13:50:08.673 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":9,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:50:08.553242+0000","last_up_change":"2023-11-03T13:50:08.553242+0000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-03T13:50:07.579756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6801","nonce":1963686469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6803","nonce":1963686469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6807","nonce":1963686469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6805","nonce":1963686469}]},"public_addr":"172.21.15.73:6801/1963686469","cluster_addr":"172.21.15.73:6803/1963686469","heartbeat_back_addr":"172.21.15.73:6807/1963686469","heartbeat_front_addr":"172.21.15.73:6805/1963686469","state":["exists","up"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6809","nonce":910736944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6811","nonce":910736944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6815","nonce":910736944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6813","nonce":910736944}]},"public_addr":"172.21.15.73:6809/910736944","cluster_addr":"172.21.15.73:6811/910736944","heartbeat_back_addr":"172.21.15.73:6815/910736944","heartbeat_front_addr":"172.21.15.73:6813/910736944","state":["exists","up"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6817","nonce":1497194653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6819","nonce":1497194653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6823","nonce":1497194653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6821","nonce":1497194653}]},"public_addr":"172.21.15.73:6817/1497194653","cluster_addr":"172.21.15.73:6819/1497194653","heartbeat_back_addr":"172.21.15.73:6823/1497194653","heartbeat_front_addr":"172.21.15.73:6821/1497194653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.494074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.432141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:50:08.684 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-03T13:50:08.684 INFO:tasks.ceph:Creating RBD pool 2023-11-03T13:50:08.685 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-03T13:50:09.566 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'rbd' created 2023-11-03T13:50:09.584 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-03T13:50:10.567 INFO:teuthology.orchestra.run.smithi073.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-03T13:50:10.583 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-03T13:50:10.583 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-03T13:50:10.584 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-03T13:50:10.584 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-03T13:50:10.584 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-03T13:50:10.905 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:10.916 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":2155676169},{"type":"v1","addr":"172.21.15.73:6825","nonce":2155676169}]},"active_addr":"172.21.15.73:6825/2155676169","active_change":"2023-11-03T13:50:03.531856+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":898465664}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":900681058}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3234126741}]}]} 2023-11-03T13:50:10.918 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-03T13:50:10.919 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-03T13:50:10.919 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-03T13:50:11.193 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:11.193 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":11,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:50:10.561477+0000","last_up_change":"2023-11-03T13:50:08.553242+0000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-03T13:50:07.579756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-03T13:50:08.976467+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6801","nonce":1963686469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6803","nonce":1963686469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6807","nonce":1963686469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6805","nonce":1963686469}]},"public_addr":"172.21.15.73:6801/1963686469","cluster_addr":"172.21.15.73:6803/1963686469","heartbeat_back_addr":"172.21.15.73:6807/1963686469","heartbeat_front_addr":"172.21.15.73:6805/1963686469","state":["exists","up"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6809","nonce":910736944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6811","nonce":910736944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6815","nonce":910736944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6813","nonce":910736944}]},"public_addr":"172.21.15.73:6809/910736944","cluster_addr":"172.21.15.73:6811/910736944","heartbeat_back_addr":"172.21.15.73:6815/910736944","heartbeat_front_addr":"172.21.15.73:6813/910736944","state":["exists","up"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6817","nonce":1497194653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6819","nonce":1497194653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6823","nonce":1497194653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6821","nonce":1497194653}]},"public_addr":"172.21.15.73:6817/1497194653","cluster_addr":"172.21.15.73:6819/1497194653","heartbeat_back_addr":"172.21.15.73:6823/1497194653","heartbeat_front_addr":"172.21.15.73:6821/1497194653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.494074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.432141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.581343+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:50:11.205 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-03T13:50:11.206 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-03T13:50:11.483 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:11.483 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":11,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:50:10.561477+0000","last_up_change":"2023-11-03T13:50:08.553242+0000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-03T13:50:07.579756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-03T13:50:08.976467+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6801","nonce":1963686469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6803","nonce":1963686469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6807","nonce":1963686469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6805","nonce":1963686469}]},"public_addr":"172.21.15.73:6801/1963686469","cluster_addr":"172.21.15.73:6803/1963686469","heartbeat_back_addr":"172.21.15.73:6807/1963686469","heartbeat_front_addr":"172.21.15.73:6805/1963686469","state":["exists","up"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6809","nonce":910736944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6811","nonce":910736944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6815","nonce":910736944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6813","nonce":910736944}]},"public_addr":"172.21.15.73:6809/910736944","cluster_addr":"172.21.15.73:6811/910736944","heartbeat_back_addr":"172.21.15.73:6815/910736944","heartbeat_front_addr":"172.21.15.73:6813/910736944","state":["exists","up"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6817","nonce":1497194653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6819","nonce":1497194653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6823","nonce":1497194653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6821","nonce":1497194653}]},"public_addr":"172.21.15.73:6817/1497194653","cluster_addr":"172.21.15.73:6819/1497194653","heartbeat_back_addr":"172.21.15.73:6823/1497194653","heartbeat_front_addr":"172.21.15.73:6821/1497194653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.494074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.432141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.581343+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:50:11.495 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-03T13:50:11.496 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-03T13:50:11.497 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-03T13:50:11.636 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2023-11-03T13:50:11.636 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-03T13:50:11.655 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2023-11-03T13:50:11.656 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-03T13:50:11.657 INFO:teuthology.orchestra.run.smithi073.stdout:38654705667 2023-11-03T13:50:11.657 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-03T13:50:11.951 INFO:teuthology.orchestra.run.smithi073.stdout:34359738371 2023-11-03T13:50:11.964 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-11-03T13:50:11.986 INFO:teuthology.orchestra.run.smithi073.stdout:34359738371 2023-11-03T13:50:11.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-11-03T13:50:12.009 INFO:teuthology.orchestra.run.smithi073.stdout:38654705666 2023-11-03T13:50:12.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2023-11-03T13:50:12.965 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-03T13:50:12.999 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-03T13:50:13.022 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-03T13:50:13.266 INFO:teuthology.orchestra.run.smithi073.stdout:34359738371 2023-11-03T13:50:13.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-11-03T13:50:13.307 INFO:teuthology.orchestra.run.smithi073.stdout:34359738371 2023-11-03T13:50:13.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-11-03T13:50:13.346 INFO:teuthology.orchestra.run.smithi073.stdout:38654705666 2023-11-03T13:50:13.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2023-11-03T13:50:14.279 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-03T13:50:14.321 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-03T13:50:14.359 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-03T13:50:14.585 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2023-11-03T13:50:14.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-11-03T13:50:14.598 DEBUG:teuthology.parallel:result is None 2023-11-03T13:50:14.619 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2023-11-03T13:50:14.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-11-03T13:50:14.633 DEBUG:teuthology.parallel:result is None 2023-11-03T13:50:14.660 INFO:teuthology.orchestra.run.smithi073.stdout:38654705667 2023-11-03T13:50:14.672 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2023-11-03T13:50:14.672 DEBUG:teuthology.parallel:result is None 2023-11-03T13:50:14.673 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-03T13:50:14.673 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:50:14.986 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:14.987 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:50:14.999 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-03T13:50:13.538855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81480,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283034040,"statfs":{"total":289910292480,"available":289826856960,"internally_reserved":0,"allocated":401408,"data_stored":1012014,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574420+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:50:11.574420+0000","last_peered":"2023-11-03T13:50:11.574420+0000","last_clean":"2023-11-03T13:50:11.574420+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:50:11.574420+0000","last_undegraded":"2023-11-03T13:50:11.574420+0000","last_fullsized":"2023-11-03T13:50:11.574420+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574050+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:50:11.574050+0000","last_peered":"2023-11-03T13:50:11.574050+0000","last_clean":"2023-11-03T13:50:11.574050+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:50:11.574050+0000","last_undegraded":"2023-11-03T13:50:11.574050+0000","last_fullsized":"2023-11-03T13:50:11.574050+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574626+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:50:11.574626+0000","last_peered":"2023-11-03T13:50:11.574626+0000","last_clean":"2023-11-03T13:50:11.574626+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:50:11.574626+0000","last_undegraded":"2023-11-03T13:50:11.574626+0000","last_fullsized":"2023-11-03T13:50:11.574626+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574481+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:50:11.574481+0000","last_peered":"2023-11-03T13:50:11.574481+0000","last_clean":"2023-11-03T13:50:11.574481+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:50:11.574481+0000","last_undegraded":"2023-11-03T13:50:11.574481+0000","last_fullsized":"2023-11-03T13:50:11.574481+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-03T13:50:10.571421+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:50:10.571421+0000","last_peered":"2023-11-03T13:50:10.571421+0000","last_clean":"2023-11-03T13:50:10.571421+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:50:10.571421+0000","last_undegraded":"2023-11-03T13:50:10.571421+0000","last_fullsized":"2023-11-03T13:50:10.571421+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:01:29.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.575723+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:50:11.575723+0000","last_peered":"2023-11-03T13:50:11.575723+0000","last_clean":"2023-11-03T13:50:11.575723+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:50:11.575723+0000","last_undegraded":"2023-11-03T13:50:11.575723+0000","last_fullsized":"2023-11-03T13:50:11.575723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.575583+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:50:11.575583+0000","last_peered":"2023-11-03T13:50:11.575583+0000","last_clean":"2023-11-03T13:50:11.575583+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:50:11.575583+0000","last_undegraded":"2023-11-03T13:50:11.575583+0000","last_fullsized":"2023-11-03T13:50:11.575583+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574723+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:50:11.574723+0000","last_peered":"2023-11-03T13:50:11.574723+0000","last_clean":"2023-11-03T13:50:11.574723+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:50:11.574723+0000","last_undegraded":"2023-11-03T13:50:11.574723+0000","last_fullsized":"2023-11-03T13:50:11.574723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.573987+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:50:11.573987+0000","last_peered":"2023-11-03T13:50:11.573987+0000","last_clean":"2023-11-03T13:50:11.573987+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:50:11.573987+0000","last_undegraded":"2023-11-03T13:50:11.573987+0000","last_fullsized":"2023-11-03T13:50:11.573987+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":98304,"data_stored":26034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":151552,"data_stored":492990,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":151552,"data_stored":492990,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:50:15.000 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:50:15.274 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:50:15.286 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-03T13:50:13.538855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81480,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283034040,"statfs":{"total":289910292480,"available":289826856960,"internally_reserved":0,"allocated":401408,"data_stored":1012014,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574420+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:50:11.574420+0000","last_peered":"2023-11-03T13:50:11.574420+0000","last_clean":"2023-11-03T13:50:11.574420+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:50:11.574420+0000","last_undegraded":"2023-11-03T13:50:11.574420+0000","last_fullsized":"2023-11-03T13:50:11.574420+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574050+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:50:11.574050+0000","last_peered":"2023-11-03T13:50:11.574050+0000","last_clean":"2023-11-03T13:50:11.574050+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:50:11.574050+0000","last_undegraded":"2023-11-03T13:50:11.574050+0000","last_fullsized":"2023-11-03T13:50:11.574050+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574626+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:50:11.574626+0000","last_peered":"2023-11-03T13:50:11.574626+0000","last_clean":"2023-11-03T13:50:11.574626+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:50:11.574626+0000","last_undegraded":"2023-11-03T13:50:11.574626+0000","last_fullsized":"2023-11-03T13:50:11.574626+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574481+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:50:11.574481+0000","last_peered":"2023-11-03T13:50:11.574481+0000","last_clean":"2023-11-03T13:50:11.574481+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:50:11.574481+0000","last_undegraded":"2023-11-03T13:50:11.574481+0000","last_fullsized":"2023-11-03T13:50:11.574481+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-03T13:50:10.571421+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:50:10.571421+0000","last_peered":"2023-11-03T13:50:10.571421+0000","last_clean":"2023-11-03T13:50:10.571421+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:50:10.571421+0000","last_undegraded":"2023-11-03T13:50:10.571421+0000","last_fullsized":"2023-11-03T13:50:10.571421+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:01:29.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.575723+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:50:11.575723+0000","last_peered":"2023-11-03T13:50:11.575723+0000","last_clean":"2023-11-03T13:50:11.575723+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:50:11.575723+0000","last_undegraded":"2023-11-03T13:50:11.575723+0000","last_fullsized":"2023-11-03T13:50:11.575723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.575583+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:50:11.575583+0000","last_peered":"2023-11-03T13:50:11.575583+0000","last_clean":"2023-11-03T13:50:11.575583+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:50:11.575583+0000","last_undegraded":"2023-11-03T13:50:11.575583+0000","last_fullsized":"2023-11-03T13:50:11.575583+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.574723+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:50:11.574723+0000","last_peered":"2023-11-03T13:50:11.574723+0000","last_clean":"2023-11-03T13:50:11.574723+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:50:11.574723+0000","last_undegraded":"2023-11-03T13:50:11.574723+0000","last_fullsized":"2023-11-03T13:50:11.574723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-03T13:50:11.573987+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:50:11.573987+0000","last_peered":"2023-11-03T13:50:11.573987+0000","last_clean":"2023-11-03T13:50:11.573987+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:50:11.573987+0000","last_undegraded":"2023-11-03T13:50:11.573987+0000","last_fullsized":"2023-11-03T13:50:11.573987+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":98304,"data_stored":26034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":151552,"data_stored":492990,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":151552,"data_stored":492990,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:50:15.288 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-03T13:50:15.289 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-03T13:50:15.289 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-03T13:50:15.290 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-03T13:50:15.600 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:50:15.601 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-03T13:50:15.613 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-03T13:50:15.614 INFO:teuthology.run_tasks:Running task workunit... 2023-11-03T13:50:15.626 INFO:tasks.workunit:Pulling workunits from ref 10145169af62d2c41c01dd8795eeddb1b869c21a 2023-11-03T13:50:15.627 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-03T13:50:15.627 INFO:tasks.workunit:timeout=3h 2023-11-03T13:50:15.627 INFO:tasks.workunit:cleanup=True 2023-11-03T13:50:15.628 DEBUG:teuthology.orchestra.run.smithi073:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-03T13:50:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:50:15.643 INFO:teuthology.orchestra.run.smithi073.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-03T13:50:15.643 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-11-03T13:50:15.698 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-11-03T13:50:15.698 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-11-03T13:50:15.752 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 10145169af62d2c41c01dd8795eeddb1b869c21a 2023-11-03T13:50:15.808 INFO:tasks.workunit.client.0.smithi073.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-03T13:51:33.134 INFO:tasks.workunit.client.0.smithi073.stderr:Note: switching to '10145169af62d2c41c01dd8795eeddb1b869c21a'. 2023-11-03T13:51:33.135 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.135 INFO:tasks.workunit.client.0.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-03T13:51:33.135 INFO:tasks.workunit.client.0.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-03T13:51:33.136 INFO:tasks.workunit.client.0.smithi073.stderr:state without impacting any branches by switching back to a branch. 2023-11-03T13:51:33.136 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.136 INFO:tasks.workunit.client.0.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-03T13:51:33.136 INFO:tasks.workunit.client.0.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-03T13:51:33.136 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.137 INFO:tasks.workunit.client.0.smithi073.stderr: git switch -c 2023-11-03T13:51:33.137 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.137 INFO:tasks.workunit.client.0.smithi073.stderr:Or undo this operation with: 2023-11-03T13:51:33.137 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.137 INFO:tasks.workunit.client.0.smithi073.stderr: git switch - 2023-11-03T13:51:33.138 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.138 INFO:tasks.workunit.client.0.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-03T13:51:33.138 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:33.138 INFO:tasks.workunit.client.0.smithi073.stderr:HEAD is now at 10145169af6 Merge branch 'wip-fix-osd-op-queue-rndmize-62171' of https://github.com/sseshasa/ceph into wip-sseshasa-testing-2023-11-02-1957 2023-11-03T13:51:33.139 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-03T13:51:33.207 INFO:tasks.workunit.client.0.smithi073.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-03T13:51:33.209 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-03T13:51:33.209 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-03T13:51:33.894 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-03T13:51:33.973 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-03T13:51:34.017 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-03T13:51:34.018 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-03T13:51:34.018 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-03T13:51:34.056 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-03T13:51:34.058 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-11-03T13:51:34.059 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-03T13:51:34.117 INFO:tasks.workunit:Running workunits matching cephtool on client.0... 2023-11-03T13:51:34.119 INFO:tasks.workunit:Running workunit cephtool/test.sh... 2023-11-03T13:51:34.119 DEBUG:teuthology.orchestra.run.smithi073:workunit test cephtool/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=10145169af62d2c41c01dd8795eeddb1b869c21a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2023-11-03T13:51:34.181 INFO:tasks.workunit.client.0.smithi073.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2023-11-03T13:51:34.185 INFO:tasks.workunit.client.0.smithi073.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh 2023-11-03T13:51:34.185 INFO:tasks.workunit.client.0.smithi073.stderr:++ TIMEOUT=300 2023-11-03T13:51:34.185 INFO:tasks.workunit.client.0.smithi073.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2023-11-03T13:51:34.186 INFO:tasks.workunit.client.0.smithi073.stderr:++ MAX_TIMEOUT=15 2023-11-03T13:51:34.186 INFO:tasks.workunit.client.0.smithi073.stderr:++ PG_NUM=4 2023-11-03T13:51:34.186 INFO:tasks.workunit.client.0.smithi073.stderr:++ TMPDIR=/tmp 2023-11-03T13:51:34.186 INFO:tasks.workunit.client.0.smithi073.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:++ TESTDIR=/home/ubuntu/cephtest 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:++ type xmlstarlet 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:++ XMLSTARLET=xmlstarlet 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:+++ uname 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:++ '[' Linux = FreeBSD ']' 2023-11-03T13:51:34.187 INFO:tasks.workunit.client.0.smithi073.stderr:++ SED=sed 2023-11-03T13:51:34.188 INFO:tasks.workunit.client.0.smithi073.stderr:++ AWK=awk 2023-11-03T13:51:34.188 INFO:tasks.workunit.client.0.smithi073.stderr:+++ stty -a 2023-11-03T13:51:34.188 INFO:tasks.workunit.client.0.smithi073.stderr:+++ head -1 2023-11-03T13:51:34.188 INFO:tasks.workunit.client.0.smithi073.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2023-11-03T13:51:34.188 INFO:tasks.workunit.client.0.smithi073.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-03T13:51:34.189 INFO:tasks.workunit.client.0.smithi073.stderr:++ termwidth= 2023-11-03T13:51:34.189 INFO:tasks.workunit.client.0.smithi073.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2023-11-03T13:51:34.189 INFO:tasks.workunit.client.0.smithi073.stderr:++ DIFFCOLOPTS='-y ' 2023-11-03T13:51:34.189 INFO:tasks.workunit.client.0.smithi073.stderr:++ KERNCORE=kernel.core_pattern 2023-11-03T13:51:34.189 INFO:tasks.workunit.client.0.smithi073.stderr:++ EXTRA_OPTS= 2023-11-03T13:51:34.191 INFO:tasks.workunit.client.0.smithi073.stderr:++ test '' = TESTS 2023-11-03T13:51:34.191 INFO:tasks.workunit.client.0.smithi073.stderr:+ set -e 2023-11-03T13:51:34.192 INFO:tasks.workunit.client.0.smithi073.stderr:+ set -o functrace 2023-11-03T13:51:34.192 INFO:tasks.workunit.client.0.smithi073.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-03T13:51:34.192 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:11: : SUDO=sudo 2023-11-03T13:51:34.192 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : export CEPH_DEV=1 2023-11-03T13:51:34.192 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:12: : CEPH_DEV=1 2023-11-03T13:51:34.193 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : mktemp -d /tmp/cephtool.XXX 2023-11-03T13:51:34.193 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:45: : TEMP_DIR=/tmp/cephtool.KY8 2023-11-03T13:51:34.193 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:46: : trap 'rm -fr /tmp/cephtool.KY8' 0 2023-11-03T13:51:34.193 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : mktemp /tmp/cephtool.KY8/test_invalid.XXX 2023-11-03T13:51:34.194 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: : TMPFILE=/tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:51:34.196 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2850: : set +x 2023-11-03T13:51:34.565 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'rbd' already exists 2023-11-03T13:51:35.672 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_injectargs 2023-11-03T13:51:35.672 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:201: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2023-11-03T13:51:35.781 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:35.790 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-11-03T13:51:35.791 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:202: test_mon_injectargs: grep false 2023-11-03T13:51:35.905 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_enable_op_tracker": "false" 2023-11-03T13:51:35.906 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:203: test_mon_injectargs: ceph tell osd.0 injectargs '--osd_enable_op_tracker --osd_op_history_duration 500' 2023-11-03T13:51:36.014 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:36.024 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-11-03T13:51:36.024 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:204: test_mon_injectargs: grep true 2023-11-03T13:51:36.138 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_enable_op_tracker": "true" 2023-11-03T13:51:36.139 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2023-11-03T13:51:36.139 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:205: test_mon_injectargs: grep 500 2023-11-03T13:51:36.252 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_op_history_duration": "500" 2023-11-03T13:51:36.253 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:206: test_mon_injectargs: ceph tell osd.0 injectargs --no-osd_enable_op_tracker 2023-11-03T13:51:36.364 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:36.373 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-11-03T13:51:36.374 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:207: test_mon_injectargs: grep false 2023-11-03T13:51:36.488 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_enable_op_tracker": "false" 2023-11-03T13:51:36.489 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:208: test_mon_injectargs: ceph tell osd.0 injectargs -- --osd_enable_op_tracker 2023-11-03T13:51:36.597 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:36.606 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-11-03T13:51:36.607 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:209: test_mon_injectargs: grep true 2023-11-03T13:51:36.721 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_enable_op_tracker": "true" 2023-11-03T13:51:36.721 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:210: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_enable_op_tracker --osd_op_history_duration 600' 2023-11-03T13:51:36.830 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:36.840 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: ceph tell osd.0 config get osd_enable_op_tracker 2023-11-03T13:51:36.840 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:211: test_mon_injectargs: grep true 2023-11-03T13:51:36.953 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_enable_op_tracker": "true" 2023-11-03T13:51:36.954 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: ceph tell osd.0 config get osd_op_history_duration 2023-11-03T13:51:36.954 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:212: test_mon_injectargs: grep 600 2023-11-03T13:51:37.323 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_op_history_duration": "600" 2023-11-03T13:51:37.323 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:214: test_mon_injectargs: ceph tell osd.0 injectargs -- '--osd_deep_scrub_interval 2419200' 2023-11-03T13:51:37.429 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:37.439 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: ceph tell osd.0 config get osd_deep_scrub_interval 2023-11-03T13:51:37.439 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:215: test_mon_injectargs: grep 2419200 2023-11-03T13:51:37.753 INFO:tasks.workunit.client.0.smithi073.stdout: "osd_deep_scrub_interval": "2419200.000000" 2023-11-03T13:51:37.753 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:217: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon_probe_timeout 2' 2023-11-03T13:51:37.862 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:37.872 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: ceph tell osd.0 config get mon_probe_timeout 2023-11-03T13:51:37.872 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:218: test_mon_injectargs: grep 2 2023-11-03T13:51:37.986 INFO:tasks.workunit.client.0.smithi073.stdout: "mon_probe_timeout": "2.000000" 2023-11-03T13:51:37.987 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:220: test_mon_injectargs: ceph tell osd.0 injectargs -- '--mon-lease 6' 2023-11-03T13:51:38.094 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:38.104 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: ceph tell osd.0 config get mon_lease 2023-11-03T13:51:38.104 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:221: test_mon_injectargs: grep 6 2023-11-03T13:51:38.389 INFO:tasks.workunit.client.0.smithi073.stderr:2023-11-03T13:51:38.387+0000 7f1ad259c640 0 --2- 172.21.15.73:0/3250866079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ad4127fa0 0x7f1ad4122b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-03T13:51:38.623 INFO:tasks.workunit.client.0.smithi073.stdout: "mon_lease": "6.000000" 2023-11-03T13:51:38.623 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:224: test_mon_injectargs: expect_false ceph tell osd.0 injectargs --osd-scrub-auto-repair-num-errors -1 2023-11-03T13:51:38.723 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:225: test_mon_injectargs: check_response 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2023-11-03T13:51:38.723 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' 2023-11-03T13:51:38.724 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:51:38.724 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:51:38.724 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:51:38.725 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'Error EINVAL: Parse error setting osd_scrub_auto_repair_num_errors to '\''-1'\'' using injectargs' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:51:38.726 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:227: test_mon_injectargs: expect_failure /tmp/cephtool.KY8 'Option --osd_op_history_duration requires an argument' ceph tell osd.0 injectargs -- --osd_op_history_duration 2023-11-03T13:51:38.727 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1917: expect_failure: local dir=/tmp/cephtool.KY8 2023-11-03T13:51:38.727 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1918: expect_failure: shift 2023-11-03T13:51:38.727 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1919: expect_failure: local 'expected=Option --osd_op_history_duration requires an argument' 2023-11-03T13:51:38.728 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1920: expect_failure: shift 2023-11-03T13:51:38.728 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1921: expect_failure: local success 2023-11-03T13:51:38.728 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 injectargs -- --osd_op_history_duration 2023-11-03T13:51:38.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1926: expect_failure: success=false 2023-11-03T13:51:38.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1929: expect_failure: false 2023-11-03T13:51:38.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet 'Option --osd_op_history_duration requires an argument' /tmp/cephtool.KY8/out 2023-11-03T13:51:38.824 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1933: expect_failure: return 0 2023-11-03T13:51:38.824 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:51:39.187 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_injectargs_SI 2023-11-03T13:51:39.187 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:39.187 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:39.188 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:39.188 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.188 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:39.294 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:39.295 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "10000" 2023-11-03T13:51:39.295 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:39.295 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:39.296 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-11-03T13:51:39.296 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:39.297 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-11-03T13:51:39.297 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-11-03T13:51:39.297 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:39.298 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-11-03T13:51:39.298 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-11-03T13:51:39.298 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:39.299 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:241: test_mon_injectargs_SI: initial_value=10000 2023-11-03T13:51:39.299 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:242: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10 2023-11-03T13:51:39.395 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:51:39.396 INFO:tasks.workunit.client.0.smithi073.stdout: "success": "mon_pg_warn_min_objects = '10' (not observed, change may require restart) " 2023-11-03T13:51:39.396 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:51:39.406 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:243: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2023-11-03T13:51:39.407 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:39.407 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:39.407 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.407 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2023-11-03T13:51:39.408 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:39.408 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:39.408 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:39.408 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.408 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:39.515 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:39.515 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "10" 2023-11-03T13:51:39.516 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:39.516 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:39.516 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2023-11-03T13:51:39.516 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:39.517 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2023-11-03T13:51:39.517 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2023-11-03T13:51:39.517 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:39.518 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2023-11-03T13:51:39.519 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2023-11-03T13:51:39.519 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:39.519 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2023-11-03T13:51:39.519 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2023-11-03T13:51:39.519 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:244: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10K 2023-11-03T13:51:39.614 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:51:39.615 INFO:tasks.workunit.client.0.smithi073.stdout: "success": "mon_pg_warn_min_objects = '10000' (not observed, change may require restart) " 2023-11-03T13:51:39.615 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:51:39.625 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:245: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2023-11-03T13:51:39.625 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:39.626 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:39.626 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.626 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2023-11-03T13:51:39.626 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:39.626 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:39.627 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:39.627 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.627 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:39.731 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:39.732 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "10000" 2023-11-03T13:51:39.732 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:39.732 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:39.733 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-11-03T13:51:39.733 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:39.734 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-11-03T13:51:39.734 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-11-03T13:51:39.734 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:39.735 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-11-03T13:51:39.735 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-11-03T13:51:39.736 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:39.736 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2023-11-03T13:51:39.736 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2023-11-03T13:51:39.736 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:246: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 1G 2023-11-03T13:51:39.832 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:51:39.832 INFO:tasks.workunit.client.0.smithi073.stdout: "success": "mon_pg_warn_min_objects = '1000000000' (not observed, change may require restart) " 2023-11-03T13:51:39.833 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:51:39.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:247: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2023-11-03T13:51:39.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:39.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:39.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.845 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2023-11-03T13:51:39.845 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:39.845 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:39.845 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:39.845 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:39.846 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:39.950 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:39.951 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "1000000000" 2023-11-03T13:51:39.951 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:39.951 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:39.951 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2023-11-03T13:51:39.951 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:39.952 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2023-11-03T13:51:39.952 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2023-11-03T13:51:39.953 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:39.954 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2023-11-03T13:51:39.954 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2023-11-03T13:51:39.954 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:39.954 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2023-11-03T13:51:39.954 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2023-11-03T13:51:39.955 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:248: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10F 2023-11-03T13:51:40.060 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:249: test_mon_injectargs_SI: check_response '(22) Invalid argument' 2023-11-03T13:51:40.061 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='(22) Invalid argument' 2023-11-03T13:51:40.061 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:51:40.061 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:51:40.061 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:51:40.062 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- '(22) Invalid argument' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:51:40.062 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:251: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10' 2023-11-03T13:51:40.158 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:40.167 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:252: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10 2023-11-03T13:51:40.168 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:40.168 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:40.168 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.168 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10 2023-11-03T13:51:40.169 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:40.169 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:40.169 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:40.169 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.169 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:40.275 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:40.275 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "10" 2023-11-03T13:51:40.275 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:40.276 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:40.276 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10"' '}' 2023-11-03T13:51:40.276 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:40.277 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10 2023-11-03T13:51:40.277 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10 2023-11-03T13:51:40.277 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:40.278 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10 2023-11-03T13:51:40.278 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10 2023-11-03T13:51:40.279 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:40.279 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10 2023-11-03T13:51:40.279 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10 != \1\0 ]] 2023-11-03T13:51:40.279 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:253: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10K' 2023-11-03T13:51:40.574 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:40.584 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:254: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 10000 2023-11-03T13:51:40.584 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:40.584 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:40.584 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.585 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=10000 2023-11-03T13:51:40.585 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:40.585 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:40.585 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:40.585 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.586 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:40.692 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:40.692 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "10000" 2023-11-03T13:51:40.693 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:40.693 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:40.693 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"10000"' '}' 2023-11-03T13:51:40.694 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:40.694 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:10000 2023-11-03T13:51:40.694 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:10000 2023-11-03T13:51:40.695 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:40.696 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=10000 2023-11-03T13:51:40.696 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 10000 2023-11-03T13:51:40.696 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:40.696 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=10000 2023-11-03T13:51:40.696 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 10000 != \1\0\0\0\0 ]] 2023-11-03T13:51:40.697 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:255: test_mon_injectargs_SI: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 1G' 2023-11-03T13:51:40.791 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:40.803 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:256: test_mon_injectargs_SI: expect_config_value mon.a mon_pg_warn_min_objects 1000000000 2023-11-03T13:51:40.804 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:140: expect_config_value: local target config_opt expected_val val 2023-11-03T13:51:40.804 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:141: expect_config_value: target=mon.a 2023-11-03T13:51:40.805 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:142: expect_config_value: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.805 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:143: expect_config_value: expected_val=1000000000 2023-11-03T13:51:40.805 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: get_config_value_or_die mon.a mon_pg_warn_min_objects 2023-11-03T13:51:40.805 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:120: get_config_value_or_die: local target config_opt raw val 2023-11-03T13:51:40.806 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:122: get_config_value_or_die: target=mon.a 2023-11-03T13:51:40.810 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:123: get_config_value_or_die: config_opt=mon_pg_warn_min_objects 2023-11-03T13:51:40.810 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: sudo ceph daemon mon.a config get mon_pg_warn_min_objects 2023-11-03T13:51:40.907 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:125: get_config_value_or_die: raw='{ 2023-11-03T13:51:40.907 INFO:tasks.workunit.client.0.smithi073.stderr: "mon_pg_warn_min_objects": "1000000000" 2023-11-03T13:51:40.907 INFO:tasks.workunit.client.0.smithi073.stderr:}' 2023-11-03T13:51:40.907 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:126: get_config_value_or_die: [[ 0 -ne 0 ]] 2023-11-03T13:51:40.908 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: echo '{' '"mon_pg_warn_min_objects":' '"1000000000"' '}' 2023-11-03T13:51:40.908 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: sed -e 's/[{} "]//g' 2023-11-03T13:51:40.909 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:131: get_config_value_or_die: raw=mon_pg_warn_min_objects:1000000000 2023-11-03T13:51:40.909 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: echo mon_pg_warn_min_objects:1000000000 2023-11-03T13:51:40.909 INFO:tasks.workunit.client.0.smithi073.stderr:///home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: cut -f2 -d: 2023-11-03T13:51:40.910 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:132: get_config_value_or_die: val=1000000000 2023-11-03T13:51:40.911 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:134: get_config_value_or_die: echo 1000000000 2023-11-03T13:51:40.911 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:135: get_config_value_or_die: return 0 2023-11-03T13:51:40.911 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:145: expect_config_value: val=1000000000 2023-11-03T13:51:40.911 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:147: expect_config_value: [[ 1000000000 != \1\0\0\0\0\0\0\0\0\0 ]] 2023-11-03T13:51:40.911 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:257: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2023-11-03T13:51:40.912 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:40.912 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_pg_warn_min_objects 10F' 2023-11-03T13:51:40.999 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: Parse error setting mon_pg_warn_min_objects to '10F' using injectargs (strict_si_cast: unit prefix not recognized). 2023-11-03T13:51:41.000 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:41.001 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:41.002 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:258: test_mon_injectargs_SI: expect_false ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2023-11-03T13:51:41.002 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:41.002 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph tell mon.a injectargs '--mon_globalid_prealloc -1' 2023-11-03T13:51:41.090 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: Parse error setting mon_globalid_prealloc to '-1' using injectargs (strict_sistrtoll: value should not be negative). 2023-11-03T13:51:41.091 INFO:tasks.workunit.client.0.smithi073.stderr: 2023-11-03T13:51:41.092 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:41.092 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:259: test_mon_injectargs_SI: sudo ceph daemon mon.a config set mon_pg_warn_min_objects 10000 2023-11-03T13:51:41.187 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:51:41.187 INFO:tasks.workunit.client.0.smithi073.stdout: "success": "mon_pg_warn_min_objects = '10000' (not observed, change may require restart) " 2023-11-03T13:51:41.187 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:51:41.196 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:51:41.559 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_1 2023-11-03T13:51:41.559 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:339: test_tiering_1: ceph osd pool create slow 2 2023-11-03T13:51:42.380 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow' already exists 2023-11-03T13:51:42.391 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:340: test_tiering_1: ceph osd pool application enable slow rados 2023-11-03T13:51:44.295 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'slow' 2023-11-03T13:51:44.309 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:341: test_tiering_1: ceph osd pool create slow2 2 2023-11-03T13:51:45.399 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow2' already exists 2023-11-03T13:51:45.410 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:342: test_tiering_1: ceph osd pool application enable slow2 rados 2023-11-03T13:51:47.296 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'slow2' 2023-11-03T13:51:47.309 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:343: test_tiering_1: ceph osd pool create cache 2 2023-11-03T13:51:48.414 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' already exists 2023-11-03T13:51:48.425 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:344: test_tiering_1: ceph osd pool create cache2 2 2023-11-03T13:51:49.422 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' already exists 2023-11-03T13:51:49.433 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:345: test_tiering_1: ceph osd tier add slow cache 2023-11-03T13:51:50.439 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' is now (or already was) a tier of 'slow' 2023-11-03T13:51:50.450 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:346: test_tiering_1: ceph osd tier add slow cache2 2023-11-03T13:51:51.442 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' is now (or already was) a tier of 'slow' 2023-11-03T13:51:51.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:347: test_tiering_1: expect_false ceph osd tier add slow2 cache 2023-11-03T13:51:51.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:51.452 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add slow2 cache 2023-11-03T13:51:51.722 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: tier pool 'cache' is already a tier of 'slow' 2023-11-03T13:51:51.725 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:51.726 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: ceph osd pool ls detail -f json 2023-11-03T13:51:51.726 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: jq '.[] | select(.pool_name == "slow") | .application_metadata["rados"]' 2023-11-03T13:51:51.726 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:349: test_tiering_1: grep '{}' 2023-11-03T13:51:52.089 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:52.090 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: ceph osd pool ls detail -f json 2023-11-03T13:51:52.090 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: jq '.[] | select(.pool_name == "slow2") | .application_metadata["rados"]' 2023-11-03T13:51:52.090 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:350: test_tiering_1: grep '{}' 2023-11-03T13:51:52.455 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:52.456 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: ceph osd pool ls detail -f json 2023-11-03T13:51:52.456 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: jq '.[] | select(.pool_name == "cache") | .application_metadata["rados"]' 2023-11-03T13:51:52.456 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:351: test_tiering_1: grep '{}' 2023-11-03T13:51:52.822 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:52.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: ceph osd pool ls detail -f json 2023-11-03T13:51:52.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: jq '.[] | select(.pool_name == "cache2") | .application_metadata["rados"]' 2023-11-03T13:51:52.823 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:352: test_tiering_1: grep '{}' 2023-11-03T13:51:53.105 INFO:tasks.workunit.client.0.smithi073.stderr:2023-11-03T13:51:53.103+0000 7fe8d1598640 0 --2- 172.21.15.73:0/621303251 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe8cc125f60 0x7fe8cc11f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-03T13:51:53.392 INFO:tasks.workunit.client.0.smithi073.stdout:{} 2023-11-03T13:51:53.393 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:354: test_tiering_1: expect_false ceph osd tier cache-mode cache forward 2023-11-03T13:51:53.393 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:53.393 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward 2023-11-03T13:51:53.662 INFO:tasks.workunit.client.0.smithi073.stderr:Invalid command: forward not in writeback|proxy|readproxy|readonly|none 2023-11-03T13:51:53.663 INFO:tasks.workunit.client.0.smithi073.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2023-11-03T13:51:53.663 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: invalid command 2023-11-03T13:51:53.664 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:53.664 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:355: test_tiering_1: expect_false ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2023-11-03T13:51:53.665 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:53.665 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache forward --yes-i-really-mean-it 2023-11-03T13:51:53.936 INFO:tasks.workunit.client.0.smithi073.stderr:Invalid command: forward not in writeback|proxy|readproxy|readonly|none 2023-11-03T13:51:53.936 INFO:tasks.workunit.client.0.smithi073.stderr:osd tier cache-mode [--yes-i-really-mean-it] : specify the caching mode for cache tier 2023-11-03T13:51:53.936 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: invalid command 2023-11-03T13:51:53.938 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:53.938 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:357: test_tiering_1: ceph osd tier cache-mode cache writeback 2023-11-03T13:51:54.451 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to writeback 2023-11-03T13:51:54.462 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:358: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly 2023-11-03T13:51:54.462 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:54.462 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly 2023-11-03T13:51:54.734 INFO:tasks.workunit.client.0.smithi073.stderr:Error EPERM: 'readonly' is not a well-supported cache mode and may corrupt your data. pass --yes-i-really-mean-it to force. 2023-11-03T13:51:54.739 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:54.739 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:359: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:51:54.739 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:51:54.739 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:51:55.011 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: unable to set cache-mode 'readonly' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2023-11-03T13:51:55.014 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:51:55.015 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:360: test_tiering_1: ceph osd tier cache-mode cache proxy 2023-11-03T13:51:55.457 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to proxy 2023-11-03T13:51:55.468 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:361: test_tiering_1: ceph osd tier cache-mode cache readproxy 2023-11-03T13:51:56.466 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to readproxy 2023-11-03T13:51:56.477 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:362: test_tiering_1: ceph osd tier cache-mode cache none 2023-11-03T13:51:57.468 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to none 2023-11-03T13:51:57.479 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:363: test_tiering_1: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:51:58.476 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to readonly 2023-11-03T13:51:58.487 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:364: test_tiering_1: ceph osd tier cache-mode cache none 2023-11-03T13:51:59.480 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to none 2023-11-03T13:51:59.491 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:365: test_tiering_1: ceph osd tier cache-mode cache writeback 2023-11-03T13:52:00.489 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to writeback 2023-11-03T13:52:00.500 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:366: test_tiering_1: ceph osd tier cache-mode cache proxy 2023-11-03T13:52:01.499 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to proxy 2023-11-03T13:52:01.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:367: test_tiering_1: ceph osd tier cache-mode cache writeback 2023-11-03T13:52:02.511 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to writeback 2023-11-03T13:52:02.522 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:368: test_tiering_1: expect_false ceph osd tier cache-mode cache none 2023-11-03T13:52:02.523 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:02.523 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache none 2023-11-03T13:52:02.794 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: unable to set cache-mode 'none' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2023-11-03T13:52:02.798 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:02.798 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:369: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:52:02.798 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:02.798 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:52:03.073 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: unable to set cache-mode 'readonly' on a 'writeback' pool; only 'proxy','readproxy' allowed. 2023-11-03T13:52:03.076 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:03.076 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:372: test_tiering_1: rados -p cache put /etc/passwd /etc/passwd 2023-11-03T13:52:03.111 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:373: test_tiering_1: flush_pg_stats 2023-11-03T13:52:03.112 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-03T13:52:03.112 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-03T13:52:03.480 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-03T13:52:03.480 INFO:tasks.workunit.client.0.smithi073.stderr:1 2023-11-03T13:52:03.480 INFO:tasks.workunit.client.0.smithi073.stderr:2' 2023-11-03T13:52:03.481 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-03T13:52:03.481 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:03.481 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-03T13:52:03.601 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738397 2023-11-03T13:52:03.601 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738397 2023-11-03T13:52:03.601 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738397' 2023-11-03T13:52:03.602 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:03.602 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-03T13:52:03.719 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738397 2023-11-03T13:52:03.719 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738397 2023-11-03T13:52:03.719 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738397 1-34359738397' 2023-11-03T13:52:03.720 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:03.720 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-03T13:52:03.842 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=38654705692 2023-11-03T13:52:03.843 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 38654705692 2023-11-03T13:52:03.843 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738397 1-34359738397 2-38654705692' 2023-11-03T13:52:03.843 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:03.843 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-34359738397 2023-11-03T13:52:03.844 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:03.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-03T13:52:03.844 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-34359738397 2023-11-03T13:52:03.845 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:03.845 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738397 2023-11-03T13:52:03.845 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 34359738397' 2023-11-03T13:52:03.846 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.0 seq 34359738397 2023-11-03T13:52:03.846 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:04.219 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738394 -lt 34359738397 2023-11-03T13:52:04.220 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-03T13:52:05.220 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-03T13:52:05.221 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:05.598 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738394 -lt 34359738397 2023-11-03T13:52:05.599 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-03T13:52:06.600 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 299 -eq 0 ']' 2023-11-03T13:52:06.600 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:06.976 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738397 -lt 34359738397 2023-11-03T13:52:06.977 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:06.977 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-34359738397 2023-11-03T13:52:06.978 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:06.978 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-03T13:52:06.979 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-34359738397 2023-11-03T13:52:06.979 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:06.979 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738397 2023-11-03T13:52:06.980 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.1 seq 34359738397 2023-11-03T13:52:06.980 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 34359738397' 2023-11-03T13:52:06.980 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-03T13:52:07.342 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738397 -lt 34359738397 2023-11-03T13:52:07.342 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:07.342 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-38654705692 2023-11-03T13:52:07.343 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:07.343 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-03T13:52:07.344 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-38654705692 2023-11-03T13:52:07.344 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:07.345 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=38654705692 2023-11-03T13:52:07.345 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.2 seq 38654705692 2023-11-03T13:52:07.346 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 38654705692' 2023-11-03T13:52:07.346 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-03T13:52:07.712 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 38654705692 -lt 38654705692 2023-11-03T13:52:07.712 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:375: test_tiering_1: ceph osd tier cache-mode cache proxy 2023-11-03T13:52:08.662 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to proxy 2023-11-03T13:52:08.673 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:376: test_tiering_1: expect_false ceph osd tier cache-mode cache none 2023-11-03T13:52:08.673 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:08.673 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache none 2023-11-03T13:52:08.942 INFO:tasks.workunit.client.0.smithi073.stderr:Error EBUSY: unable to set cache-mode 'none' on pool 'cache': dirty objects found 2023-11-03T13:52:08.946 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:08.946 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:377: test_tiering_1: expect_false ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:52:08.946 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:08.946 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:52:09.215 INFO:tasks.workunit.client.0.smithi073.stderr:Error EBUSY: unable to set cache-mode 'readonly' on pool 'cache': dirty objects found 2023-11-03T13:52:09.219 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:09.219 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:378: test_tiering_1: ceph osd tier cache-mode cache writeback 2023-11-03T13:52:09.666 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to writeback 2023-11-03T13:52:09.677 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:380: test_tiering_1: rados -p cache rm /etc/passwd 2023-11-03T13:52:09.710 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:381: test_tiering_1: rados -p cache cache-flush-evict-all 2023-11-03T13:52:09.739 INFO:tasks.workunit.client.0.smithi073.stdout: /etc/passwd 2023-11-03T13:52:09.747 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:382: test_tiering_1: flush_pg_stats 2023-11-03T13:52:09.747 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-03T13:52:09.748 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-03T13:52:10.115 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-03T13:52:10.116 INFO:tasks.workunit.client.0.smithi073.stderr:1 2023-11-03T13:52:10.116 INFO:tasks.workunit.client.0.smithi073.stderr:2' 2023-11-03T13:52:10.116 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-03T13:52:10.116 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:10.117 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-03T13:52:10.231 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738400 2023-11-03T13:52:10.231 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738400 2023-11-03T13:52:10.232 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738400' 2023-11-03T13:52:10.232 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:10.232 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-03T13:52:10.348 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738400 2023-11-03T13:52:10.348 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738400 2023-11-03T13:52:10.349 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738400 1-34359738400' 2023-11-03T13:52:10.349 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:52:10.349 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-03T13:52:10.465 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=38654705695 2023-11-03T13:52:10.466 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 38654705695 2023-11-03T13:52:10.466 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738400 1-34359738400 2-38654705695' 2023-11-03T13:52:10.466 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:10.467 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-34359738400 2023-11-03T13:52:10.467 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:10.467 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-03T13:52:10.468 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-34359738400 2023-11-03T13:52:10.468 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:10.469 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738400 2023-11-03T13:52:10.469 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 34359738400' 2023-11-03T13:52:10.469 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.0 seq 34359738400 2023-11-03T13:52:10.470 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:10.833 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738398 -lt 34359738400 2023-11-03T13:52:10.834 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-03T13:52:11.835 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-03T13:52:11.835 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:12.204 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738398 -lt 34359738400 2023-11-03T13:52:12.204 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-03T13:52:13.205 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 299 -eq 0 ']' 2023-11-03T13:52:13.206 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:52:13.573 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738400 -lt 34359738400 2023-11-03T13:52:13.573 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:13.574 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-34359738400 2023-11-03T13:52:13.574 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:13.575 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-03T13:52:13.575 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-34359738400 2023-11-03T13:52:13.575 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:13.576 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738400 2023-11-03T13:52:13.576 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.1 seq 34359738400 2023-11-03T13:52:13.577 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 34359738400' 2023-11-03T13:52:13.577 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-03T13:52:13.939 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738400 -lt 34359738400 2023-11-03T13:52:13.940 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:52:13.941 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-38654705695 2023-11-03T13:52:13.941 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:52:13.942 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-03T13:52:13.942 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-38654705695 2023-11-03T13:52:13.942 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:52:13.943 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=38654705695 2023-11-03T13:52:13.944 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.2 seq 38654705695 2023-11-03T13:52:13.944 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 38654705695' 2023-11-03T13:52:13.944 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-03T13:52:14.310 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 38654705695 -lt 38654705695 2023-11-03T13:52:14.311 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:384: test_tiering_1: ceph osd tier cache-mode cache proxy 2023-11-03T13:52:15.311 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to proxy 2023-11-03T13:52:15.322 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:385: test_tiering_1: ceph osd tier cache-mode cache none 2023-11-03T13:52:16.319 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to none 2023-11-03T13:52:16.330 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:386: test_tiering_1: ceph osd tier cache-mode cache readonly --yes-i-really-mean-it 2023-11-03T13:52:17.326 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to readonly 2023-11-03T13:52:17.336 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:387: test_tiering_1: TRIES=0 2023-11-03T13:52:17.337 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:388: test_tiering_1: ceph osd pool set cache pg_num 3 --yes-i-really-mean-it 2023-11-03T13:52:19.247 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:395: test_tiering_1: expect_false ceph osd pool set cache pg_num 4 2023-11-03T13:52:19.247 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:19.247 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd pool set cache pg_num 4 2023-11-03T13:52:19.506 INFO:tasks.workunit.client.0.smithi073.stderr:Error EPERM: splits in cache pools must be followed by scrubs and leave sufficient free space to avoid overfilling. use --yes-i-really-mean-it to force. 2023-11-03T13:52:19.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:19.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:396: test_tiering_1: ceph osd tier cache-mode cache none 2023-11-03T13:52:20.346 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache' to none 2023-11-03T13:52:20.357 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:397: test_tiering_1: ceph osd tier set-overlay slow cache 2023-11-03T13:52:21.356 INFO:tasks.workunit.client.0.smithi073.stderr:overlay for 'slow' is now (or already was) 'cache' 2023-11-03T13:52:21.368 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:398: test_tiering_1: expect_false ceph osd tier set-overlay slow cache2 2023-11-03T13:52:21.368 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:21.368 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier set-overlay slow cache2 2023-11-03T13:52:21.640 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: pool 'slow' has overlay 'cache'; please remove-overlay first 2023-11-03T13:52:21.643 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:21.644 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:399: test_tiering_1: expect_false ceph osd tier remove slow cache 2023-11-03T13:52:21.644 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:21.644 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier remove slow cache 2023-11-03T13:52:21.914 INFO:tasks.workunit.client.0.smithi073.stderr:Error EBUSY: tier pool 'cache' is the overlay for 'slow'; please remove-overlay first 2023-11-03T13:52:21.918 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:21.918 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:400: test_tiering_1: ceph osd tier remove-overlay slow 2023-11-03T13:52:22.362 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'slow' 2023-11-03T13:52:22.373 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:401: test_tiering_1: ceph osd tier set-overlay slow cache2 2023-11-03T13:52:23.366 INFO:tasks.workunit.client.0.smithi073.stderr:overlay for 'slow' is now (or already was) 'cache2' 2023-11-03T13:52:23.377 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:402: test_tiering_1: ceph osd tier remove-overlay slow 2023-11-03T13:52:24.372 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'slow' 2023-11-03T13:52:24.384 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:403: test_tiering_1: ceph osd tier remove slow cache 2023-11-03T13:52:25.376 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' is now (or already was) not a tier of 'slow' 2023-11-03T13:52:25.387 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:404: test_tiering_1: ceph osd tier add slow2 cache 2023-11-03T13:52:26.379 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' is now (or already was) a tier of 'slow2' 2023-11-03T13:52:26.390 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:405: test_tiering_1: expect_false ceph osd tier set-overlay slow cache 2023-11-03T13:52:26.390 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:26.391 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier set-overlay slow cache 2023-11-03T13:52:26.661 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: tier pool 'cache' is not a tier of 'slow' 2023-11-03T13:52:26.664 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:26.665 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:406: test_tiering_1: ceph osd tier set-overlay slow2 cache 2023-11-03T13:52:27.390 INFO:tasks.workunit.client.0.smithi073.stderr:overlay for 'slow2' is now (or already was) 'cache' 2023-11-03T13:52:27.401 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:407: test_tiering_1: ceph osd tier remove-overlay slow2 2023-11-03T13:52:28.407 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'slow2' 2023-11-03T13:52:28.418 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:408: test_tiering_1: ceph osd tier remove slow2 cache 2023-11-03T13:52:29.413 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' is now (or already was) not a tier of 'slow2' 2023-11-03T13:52:29.423 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:409: test_tiering_1: ceph osd tier remove slow cache2 2023-11-03T13:52:30.413 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' is now (or already was) not a tier of 'slow' 2023-11-03T13:52:30.424 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:412: test_tiering_1: rados -p cache2 put /etc/passwd /etc/passwd 2023-11-03T13:52:30.662 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2023-11-03T13:52:30.663 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2023-11-03T13:52:30.663 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2023-11-03T13:52:31.149 INFO:tasks.workunit.client.0.smithi073.stdout:waiting for pg stats to flush 2023-11-03T13:52:31.150 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:414: test_tiering_1: echo waiting for pg stats to flush 2023-11-03T13:52:31.150 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:415: test_tiering_1: sleep 2 2023-11-03T13:52:33.151 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2023-11-03T13:52:33.152 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2023-11-03T13:52:33.152 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2023-11-03T13:52:33.654 INFO:tasks.workunit.client.0.smithi073.stdout:waiting for pg stats to flush 2023-11-03T13:52:33.654 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:414: test_tiering_1: echo waiting for pg stats to flush 2023-11-03T13:52:33.654 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:415: test_tiering_1: sleep 2 2023-11-03T13:52:35.655 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: ceph df 2023-11-03T13:52:35.656 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep cache2 2023-11-03T13:52:35.656 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:413: test_tiering_1: grep ' 1 ' 2023-11-03T13:52:36.151 INFO:tasks.workunit.client.0.smithi073.stdout:cache2 6 2 5.5 KiB 1 8 KiB 0 128 GiB 2023-11-03T13:52:36.151 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:417: test_tiering_1: expect_false ceph osd tier add slow cache2 2023-11-03T13:52:36.152 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:36.152 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add slow cache2 2023-11-03T13:52:36.424 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOTEMPTY: tier pool 'cache2' is not empty; --force-nonempty to force 2023-11-03T13:52:36.428 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:36.428 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:418: test_tiering_1: ceph osd tier add slow cache2 --force-nonempty 2023-11-03T13:52:37.434 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' is now (or already was) a tier of 'slow' 2023-11-03T13:52:37.445 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:419: test_tiering_1: ceph osd tier remove slow cache2 2023-11-03T13:52:38.436 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' is now (or already was) not a tier of 'slow' 2023-11-03T13:52:38.447 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:421: test_tiering_1: ceph osd pool ls 2023-11-03T13:52:38.448 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:421: test_tiering_1: grep cache2 2023-11-03T13:52:38.810 INFO:tasks.workunit.client.0.smithi073.stdout:cache2 2023-11-03T13:52:38.811 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:422: test_tiering_1: ceph osd pool ls -f json-pretty 2023-11-03T13:52:38.811 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:422: test_tiering_1: grep cache2 2023-11-03T13:52:39.178 INFO:tasks.workunit.client.0.smithi073.stdout: "cache2" 2023-11-03T13:52:39.179 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:423: test_tiering_1: ceph osd pool ls detail 2023-11-03T13:52:39.179 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:423: test_tiering_1: grep cache2 2023-11-03T13:52:39.546 INFO:tasks.workunit.client.0.smithi073.stdout:pool 6 'cache2' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 51 lfor 43/43/43 flags hashpspool stripe_width 0 application rados read_balance_score 1.50 2023-11-03T13:52:39.547 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:424: test_tiering_1: ceph osd pool ls detail -f json-pretty 2023-11-03T13:52:39.547 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:424: test_tiering_1: grep cache2 2023-11-03T13:52:39.908 INFO:tasks.workunit.client.0.smithi073.stdout: "pool_name": "cache2", 2023-11-03T13:52:39.908 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:426: test_tiering_1: ceph osd pool delete slow slow --yes-i-really-really-mean-it 2023-11-03T13:52:40.429 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow' does not exist 2023-11-03T13:52:40.440 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:427: test_tiering_1: ceph osd pool delete slow2 slow2 --yes-i-really-really-mean-it 2023-11-03T13:52:41.433 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow2' does not exist 2023-11-03T13:52:41.444 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:428: test_tiering_1: ceph osd pool delete cache cache --yes-i-really-really-mean-it 2023-11-03T13:52:42.437 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache' does not exist 2023-11-03T13:52:42.448 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:429: test_tiering_1: ceph osd pool delete cache2 cache2 --yes-i-really-really-mean-it 2023-11-03T13:52:43.440 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache2' does not exist 2023-11-03T13:52:43.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:52:43.815 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_2 2023-11-03T13:52:43.815 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:435: test_tiering_2: ceph osd pool create snap_base 2 2023-11-03T13:52:44.447 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'snap_base' already exists 2023-11-03T13:52:44.458 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:436: test_tiering_2: ceph osd pool application enable snap_base rados 2023-11-03T13:52:46.356 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'snap_base' 2023-11-03T13:52:46.371 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:437: test_tiering_2: ceph osd pool create snap_cache 2 2023-11-03T13:52:47.465 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'snap_cache' already exists 2023-11-03T13:52:47.477 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:438: test_tiering_2: ceph osd pool mksnap snap_cache snapname 2023-11-03T13:52:48.465 INFO:tasks.workunit.client.0.smithi073.stderr:pool snap_cache snap snapname already exists 2023-11-03T13:52:48.476 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:439: test_tiering_2: expect_false ceph osd tier add snap_base snap_cache 2023-11-03T13:52:48.476 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:48.476 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add snap_base snap_cache 2023-11-03T13:52:48.745 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOTEMPTY: tier pool 'snap_cache' has snapshot state; it cannot be added as a tier without breaking the pool 2023-11-03T13:52:48.749 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:48.749 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:440: test_tiering_2: ceph osd pool delete snap_base snap_base --yes-i-really-really-mean-it 2023-11-03T13:52:49.476 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'snap_base' does not exist 2023-11-03T13:52:49.487 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:441: test_tiering_2: ceph osd pool delete snap_cache snap_cache --yes-i-really-really-mean-it 2023-11-03T13:52:50.491 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'snap_cache' does not exist 2023-11-03T13:52:50.502 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:52:50.867 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_3 2023-11-03T13:52:50.868 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:447: test_tiering_3: ceph osd pool create basex 2 2023-11-03T13:52:51.496 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basex' already exists 2023-11-03T13:52:51.507 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:448: test_tiering_3: ceph osd pool application enable basex rados 2023-11-03T13:52:53.397 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'basex' 2023-11-03T13:52:53.412 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:449: test_tiering_3: ceph osd pool create cachex 2 2023-11-03T13:52:54.517 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachex' already exists 2023-11-03T13:52:54.529 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:450: test_tiering_3: ceph osd tier add basex cachex 2023-11-03T13:52:55.544 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachex' is now (or already was) a tier of 'basex' 2023-11-03T13:52:55.554 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:451: test_tiering_3: expect_false ceph osd pool mksnap cache snapname 2023-11-03T13:52:55.555 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:52:55.555 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd pool mksnap cache snapname 2023-11-03T13:52:55.811 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOENT: unrecognized pool 'cache' 2023-11-03T13:52:55.815 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:52:55.815 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:452: test_tiering_3: ceph osd tier remove basex cachex 2023-11-03T13:52:56.544 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachex' is now (or already was) not a tier of 'basex' 2023-11-03T13:52:56.555 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:453: test_tiering_3: ceph osd pool delete basex basex --yes-i-really-really-mean-it 2023-11-03T13:52:57.532 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basex' does not exist 2023-11-03T13:52:57.542 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:454: test_tiering_3: ceph osd pool delete cachex cachex --yes-i-really-really-mean-it 2023-11-03T13:52:58.537 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachex' does not exist 2023-11-03T13:52:58.547 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:52:58.910 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_4 2023-11-03T13:52:58.911 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:460: test_tiering_4: ceph osd pool create eccache 2 2 erasure 2023-11-03T13:53:00.561 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'eccache' already exists 2023-11-03T13:53:00.572 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:461: test_tiering_4: expect_false ceph osd set-require-min-compat-client bobtail 2023-11-03T13:53:00.573 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:53:00.573 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd set-require-min-compat-client bobtail 2023-11-03T13:53:00.833 INFO:tasks.workunit.client.0.smithi073.stderr:Error EPERM: osdmap current utilizes features that require jewel; cannot set require_min_compat_client below that to bobtail 2023-11-03T13:53:00.837 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:53:00.837 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:462: test_tiering_4: ceph osd pool create repbase 2 2023-11-03T13:53:01.561 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'repbase' already exists 2023-11-03T13:53:01.572 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:463: test_tiering_4: ceph osd pool application enable repbase rados 2023-11-03T13:53:03.455 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'repbase' 2023-11-03T13:53:03.469 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:464: test_tiering_4: expect_false ceph osd tier add repbase eccache 2023-11-03T13:53:03.469 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:53:03.470 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph osd tier add repbase eccache 2023-11-03T13:53:03.746 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOTSUP: tier pool 'eccache' is an ec pool, which cannot be a tier 2023-11-03T13:53:03.750 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:53:03.750 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:465: test_tiering_4: ceph osd pool delete repbase repbase --yes-i-really-really-mean-it 2023-11-03T13:53:04.573 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'repbase' does not exist 2023-11-03T13:53:04.584 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:466: test_tiering_4: ceph osd pool delete eccache eccache --yes-i-really-really-mean-it 2023-11-03T13:53:05.581 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'eccache' does not exist 2023-11-03T13:53:05.592 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:53:05.956 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_5 2023-11-03T13:53:05.956 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:472: test_tiering_5: ceph osd pool create slow 2 2023-11-03T13:53:06.581 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow' already exists 2023-11-03T13:53:06.592 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:473: test_tiering_5: ceph osd pool application enable slow rados 2023-11-03T13:53:08.491 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'slow' 2023-11-03T13:53:08.506 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:474: test_tiering_5: ceph osd pool create cache3 2 2023-11-03T13:53:09.598 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache3' already exists 2023-11-03T13:53:09.609 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:475: test_tiering_5: ceph osd tier add-cache slow cache3 1024000 2023-11-03T13:53:10.617 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache3' is now (or already was) a tier of 'slow' 2023-11-03T13:53:10.628 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: ceph osd dump 2023-11-03T13:53:10.629 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep cache3 2023-11-03T13:53:10.629 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep bloom 2023-11-03T13:53:10.629 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep 'false_positive_probability: 0.05' 2023-11-03T13:53:10.629 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep 'target_bytes 1024000' 2023-11-03T13:53:10.630 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:476: test_tiering_5: grep '1200s x4' 2023-11-03T13:53:10.996 INFO:tasks.workunit.client.0.smithi073.stdout:pool 14 'cache3' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 82 lfor 82/82/82 flags hashpspool,creating tier_of 13 cache_mode writeback target_bytes 1024000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 1200s x4 decay_rate 20 search_last_n 1 min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0 application rados read_balance_score 1.50 2023-11-03T13:53:10.996 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:477: test_tiering_5: ceph osd tier remove slow cache3 2023-11-03T13:53:11.267 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:477: test_tiering_5: true 2023-11-03T13:53:11.267 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:478: test_tiering_5: check_response 'EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' 2023-11-03T13:53:11.268 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' 2023-11-03T13:53:11.268 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:53:11.268 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:53:11.268 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:53:11.268 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: tier pool '\''cache3'\'' is the overlay for '\''slow'\''; please remove-overlay first' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:53:11.269 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:479: test_tiering_5: ceph osd tier remove-overlay slow 2023-11-03T13:53:12.637 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'slow' 2023-11-03T13:53:12.650 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:480: test_tiering_5: ceph osd tier remove slow cache3 2023-11-03T13:53:13.644 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache3' is now (or already was) not a tier of 'slow' 2023-11-03T13:53:13.655 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:481: test_tiering_5: ceph osd pool ls 2023-11-03T13:53:13.655 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:481: test_tiering_5: grep cache3 2023-11-03T13:53:14.015 INFO:tasks.workunit.client.0.smithi073.stdout:cache3 2023-11-03T13:53:14.016 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:482: test_tiering_5: ceph osd pool delete cache3 cache3 --yes-i-really-really-mean-it 2023-11-03T13:53:14.648 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache3' does not exist 2023-11-03T13:53:14.660 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:483: test_tiering_5: ceph osd pool ls 2023-11-03T13:53:14.660 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:483: test_tiering_5: grep cache3 2023-11-03T13:53:15.024 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:484: test_tiering_5: ceph osd pool delete slow slow --yes-i-really-really-mean-it 2023-11-03T13:53:15.649 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'slow' does not exist 2023-11-03T13:53:15.661 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:53:16.019 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_6 2023-11-03T13:53:16.020 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:490: test_tiering_6: ceph osd pool create datapool 2 2023-11-03T13:53:16.653 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' already exists 2023-11-03T13:53:16.664 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:491: test_tiering_6: ceph osd pool application enable datapool rados 2023-11-03T13:53:18.560 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'datapool' 2023-11-03T13:53:18.572 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:492: test_tiering_6: ceph osd pool create cachepool 2 2023-11-03T13:53:19.665 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' already exists 2023-11-03T13:53:19.676 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:493: test_tiering_6: ceph osd tier add-cache datapool cachepool 1024000 2023-11-03T13:53:20.680 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' is now (or already was) a tier of 'datapool' 2023-11-03T13:53:20.691 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:494: test_tiering_6: ceph osd tier cache-mode cachepool writeback 2023-11-03T13:53:21.076 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cachepool' to writeback 2023-11-03T13:53:21.086 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:495: test_tiering_6: rados -p datapool put object /etc/passwd 2023-11-03T13:53:21.123 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:496: test_tiering_6: rados -p cachepool stat object 2023-11-03T13:53:21.153 INFO:tasks.workunit.client.0.smithi073.stdout:cachepool/object mtime 2023-11-03T13:53:21.000000+0000, size 11261 2023-11-03T13:53:21.155 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:497: test_tiering_6: rados -p cachepool cache-flush object 2023-11-03T13:53:21.187 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:498: test_tiering_6: rados -p datapool stat object 2023-11-03T13:53:21.419 INFO:tasks.workunit.client.0.smithi073.stdout:datapool/object mtime 2023-11-03T13:53:21.000000+0000, size 11261 2023-11-03T13:53:21.421 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:499: test_tiering_6: ceph osd tier remove-overlay datapool 2023-11-03T13:53:22.704 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'datapool' 2023-11-03T13:53:22.714 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:500: test_tiering_6: ceph osd tier remove datapool cachepool 2023-11-03T13:53:23.710 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' is now (or already was) not a tier of 'datapool' 2023-11-03T13:53:23.721 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:501: test_tiering_6: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2023-11-03T13:53:24.701 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' does not exist 2023-11-03T13:53:24.711 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:502: test_tiering_6: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2023-11-03T13:53:25.713 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' does not exist 2023-11-03T13:53:25.724 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:53:26.087 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_7 2023-11-03T13:53:26.087 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:508: test_tiering_7: ceph osd pool create datapool 2 2023-11-03T13:53:26.711 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' already exists 2023-11-03T13:53:26.722 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:509: test_tiering_7: ceph osd pool application enable datapool rados 2023-11-03T13:53:28.616 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'datapool' 2023-11-03T13:53:28.629 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:510: test_tiering_7: ceph osd pool create cachepool 2 2023-11-03T13:53:29.727 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' already exists 2023-11-03T13:53:29.738 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:511: test_tiering_7: ceph osd tier add-cache datapool cachepool 1024000 2023-11-03T13:53:30.747 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' is now (or already was) a tier of 'datapool' 2023-11-03T13:53:30.758 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:512: test_tiering_7: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2023-11-03T13:53:31.020 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:512: test_tiering_7: true 2023-11-03T13:53:31.020 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:513: test_tiering_7: check_response 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' 2023-11-03T13:53:31.020 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' 2023-11-03T13:53:31.020 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:53:31.021 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:53:31.021 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:53:31.021 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:53:31.022 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:514: test_tiering_7: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2023-11-03T13:53:31.282 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:514: test_tiering_7: true 2023-11-03T13:53:31.283 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:515: test_tiering_7: check_response 'EBUSY: pool '\''datapool'\'' has tiers cachepool' 2023-11-03T13:53:31.283 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EBUSY: pool '\''datapool'\'' has tiers cachepool' 2023-11-03T13:53:31.283 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:53:31.283 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:53:31.283 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:53:31.284 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EBUSY: pool '\''datapool'\'' has tiers cachepool' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:53:31.284 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:516: test_tiering_7: ceph osd tier remove-overlay datapool 2023-11-03T13:53:31.751 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'datapool' 2023-11-03T13:53:31.762 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:517: test_tiering_7: ceph osd tier remove datapool cachepool 2023-11-03T13:53:32.754 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' is now (or already was) not a tier of 'datapool' 2023-11-03T13:53:32.764 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:518: test_tiering_7: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it 2023-11-03T13:53:33.741 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cachepool' does not exist 2023-11-03T13:53:33.752 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:519: test_tiering_7: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2023-11-03T13:53:34.763 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' does not exist 2023-11-03T13:53:34.774 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:53:35.136 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_8 2023-11-03T13:53:35.137 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:525: test_tiering_8: ceph osd set notieragent 2023-11-03T13:53:36.670 INFO:tasks.workunit.client.0.smithi073.stderr:notieragent is set 2023-11-03T13:53:36.683 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:526: test_tiering_8: ceph osd pool create datapool 2 2023-11-03T13:53:37.777 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' already exists 2023-11-03T13:53:37.788 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:527: test_tiering_8: ceph osd pool application enable datapool rados 2023-11-03T13:53:39.685 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'datapool' 2023-11-03T13:53:39.700 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:528: test_tiering_8: ceph osd pool create cache4 2 2023-11-03T13:53:40.802 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache4' already exists 2023-11-03T13:53:40.812 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:529: test_tiering_8: ceph osd tier add-cache datapool cache4 1024000 2023-11-03T13:53:41.825 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache4' is now (or already was) a tier of 'datapool' 2023-11-03T13:53:41.836 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:530: test_tiering_8: ceph osd tier cache-mode cache4 writeback 2023-11-03T13:53:42.220 INFO:tasks.workunit.client.0.smithi073.stderr:set cache-mode for pool 'cache4' to writeback 2023-11-03T13:53:42.231 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: mktemp 2023-11-03T13:53:42.232 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: grep tmp 2023-11-03T13:53:42.233 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:531: test_tiering_8: tmpfile=/tmp/tmp.liopGzDjXE 2023-11-03T13:53:42.233 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:532: test_tiering_8: dd if=/dev/zero of=/tmp/tmp.liopGzDjXE bs=4K count=1 2023-11-03T13:53:42.234 INFO:tasks.workunit.client.0.smithi073.stderr:1+0 records in 2023-11-03T13:53:42.234 INFO:tasks.workunit.client.0.smithi073.stderr:1+0 records out 2023-11-03T13:53:42.234 INFO:tasks.workunit.client.0.smithi073.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.1871e-05 s, 57.0 MB/s 2023-11-03T13:53:42.234 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:533: test_tiering_8: ceph osd pool set cache4 target_max_objects 200 2023-11-03T13:53:43.721 INFO:tasks.workunit.client.0.smithi073.stderr:set pool 20 target_max_objects to 200 2023-11-03T13:53:43.741 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:534: test_tiering_8: ceph osd pool set cache4 target_max_bytes 1000000 2023-11-03T13:53:45.745 INFO:tasks.workunit.client.0.smithi073.stderr:set pool 20 target_max_bytes to 1000000 2023-11-03T13:53:45.759 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:535: test_tiering_8: rados -p cache4 put foo1 /tmp/tmp.liopGzDjXE 2023-11-03T13:53:45.797 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:536: test_tiering_8: rados -p cache4 put foo2 /tmp/tmp.liopGzDjXE 2023-11-03T13:53:46.028 INFO:tasks.workunit.client.0.smithi073.stderr:2023-11-03T13:53:46.026+0000 7f93eb108640 0 --2- 172.21.15.73:0/287543355 >> [v2:172.21.15.73:3302/0,v1:172.21.15.73:6791/0] conn(0x560fa4df6990 0x560fa4def8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-03T13:53:46.034 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:537: test_tiering_8: rm -f /tmp/tmp.liopGzDjXE 2023-11-03T13:53:46.035 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:538: test_tiering_8: flush_pg_stats 2023-11-03T13:53:46.035 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-03T13:53:46.035 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-03T13:53:46.403 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-03T13:53:46.403 INFO:tasks.workunit.client.0.smithi073.stderr:1 2023-11-03T13:53:46.403 INFO:tasks.workunit.client.0.smithi073.stderr:2' 2023-11-03T13:53:46.404 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-03T13:53:46.404 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:53:46.404 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-03T13:53:46.520 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738421 2023-11-03T13:53:46.520 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738421 2023-11-03T13:53:46.521 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738421' 2023-11-03T13:53:46.521 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:53:46.521 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-03T13:53:46.638 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=34359738421 2023-11-03T13:53:46.638 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 34359738421 2023-11-03T13:53:46.638 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738421 1-34359738421' 2023-11-03T13:53:46.638 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-03T13:53:46.638 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-03T13:53:46.756 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=38654705719 2023-11-03T13:53:46.756 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 38654705719 2023-11-03T13:53:46.756 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-34359738421 1-34359738421 2-38654705719' 2023-11-03T13:53:46.757 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:53:46.757 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-34359738421 2023-11-03T13:53:46.757 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:53:46.758 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-03T13:53:46.758 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-34359738421 2023-11-03T13:53:46.759 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:53:46.759 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738421 2023-11-03T13:53:46.759 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 34359738421' 2023-11-03T13:53:46.760 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.0 seq 34359738421 2023-11-03T13:53:46.760 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:53:47.122 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738419 -lt 34359738421 2023-11-03T13:53:47.122 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-03T13:53:48.123 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-03T13:53:48.124 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-03T13:53:48.486 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738421 -lt 34359738421 2023-11-03T13:53:48.486 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:53:48.487 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-34359738421 2023-11-03T13:53:48.487 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:53:48.488 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-03T13:53:48.488 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-34359738421 2023-11-03T13:53:48.488 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:53:48.489 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=34359738421 2023-11-03T13:53:48.489 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 34359738421' 2023-11-03T13:53:48.490 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.1 seq 34359738421 2023-11-03T13:53:48.490 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-03T13:53:48.856 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 34359738421 -lt 34359738421 2023-11-03T13:53:48.856 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-03T13:53:48.857 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-38654705719 2023-11-03T13:53:48.857 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-03T13:53:48.857 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-03T13:53:48.858 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-38654705719 2023-11-03T13:53:48.858 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-03T13:53:48.859 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=38654705719 2023-11-03T13:53:48.859 INFO:tasks.workunit.client.0.smithi073.stdout:waiting osd.2 seq 38654705719 2023-11-03T13:53:48.860 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 38654705719' 2023-11-03T13:53:48.860 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-03T13:53:49.226 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:2169: flush_pg_stats: test 38654705719 -lt 38654705719 2023-11-03T13:53:49.226 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: ceph df 2023-11-03T13:53:49.226 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: grep datapool 2023-11-03T13:53:49.227 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:539: test_tiering_8: grep ' 2 ' 2023-11-03T13:53:49.713 INFO:tasks.workunit.client.0.smithi073.stdout:datapool 19 2 0 B 2 0 B 0 128 GiB 2023-11-03T13:53:49.714 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:540: test_tiering_8: ceph osd tier remove-overlay datapool 2023-11-03T13:53:50.314 INFO:tasks.workunit.client.0.smithi073.stderr:there is now (or already was) no overlay for 'datapool' 2023-11-03T13:53:50.325 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:541: test_tiering_8: ceph osd tier remove datapool cache4 2023-11-03T13:53:51.317 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache4' is now (or already was) not a tier of 'datapool' 2023-11-03T13:53:51.328 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:542: test_tiering_8: ceph osd pool delete cache4 cache4 --yes-i-really-really-mean-it 2023-11-03T13:53:52.311 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache4' does not exist 2023-11-03T13:53:52.322 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:543: test_tiering_8: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it 2023-11-03T13:53:53.313 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'datapool' does not exist 2023-11-03T13:53:53.324 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:544: test_tiering_8: ceph osd unset notieragent 2023-11-03T13:53:55.223 INFO:tasks.workunit.client.0.smithi073.stderr:notieragent is unset 2023-11-03T13:53:55.238 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:53:55.611 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_tiering_9 2023-11-03T13:53:55.612 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:553: test_tiering_9: ceph osd pool create basepoolA 2 2023-11-03T13:53:56.335 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basepoolA' already exists 2023-11-03T13:53:56.346 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:554: test_tiering_9: ceph osd pool application enable basepoolA rados 2023-11-03T13:53:58.250 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'basepoolA' 2023-11-03T13:53:58.262 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:555: test_tiering_9: ceph osd pool create basepoolB 2 2023-11-03T13:53:59.353 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basepoolB' already exists 2023-11-03T13:53:59.364 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:556: test_tiering_9: ceph osd pool application enable basepoolB rados 2023-11-03T13:54:01.251 INFO:tasks.workunit.client.0.smithi073.stderr:enabled application 'rados' on pool 'basepoolB' 2023-11-03T13:54:01.266 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: ceph osd dump 2023-11-03T13:54:01.266 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: grep 'pool.*basepoolA' 2023-11-03T13:54:01.266 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: awk '{print $2;}' 2023-11-03T13:54:01.637 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:557: test_tiering_9: poolA_id=21 2023-11-03T13:54:01.638 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: ceph osd dump 2023-11-03T13:54:01.638 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: grep 'pool.*basepoolB' 2023-11-03T13:54:01.639 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: awk '{print $2;}' 2023-11-03T13:54:02.004 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:558: test_tiering_9: poolB_id=22 2023-11-03T13:54:02.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:560: test_tiering_9: ceph osd pool create cache5 2 2023-11-03T13:54:02.430 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache5' already exists 2023-11-03T13:54:02.441 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:561: test_tiering_9: ceph osd pool create cache6 2 2023-11-03T13:54:03.432 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache6' already exists 2023-11-03T13:54:03.442 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:562: test_tiering_9: ceph osd tier add basepoolA cache5 2023-11-03T13:54:04.453 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache5' is now (or already was) a tier of 'basepoolA' 2023-11-03T13:54:04.464 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:563: test_tiering_9: ceph osd tier add basepoolB cache6 2023-11-03T13:54:05.455 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache6' is now (or already was) a tier of 'basepoolB' 2023-11-03T13:54:05.466 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:564: test_tiering_9: ceph osd tier remove basepoolB cache5 2023-11-03T13:54:05.467 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:564: test_tiering_9: grep 'not a tier of' 2023-11-03T13:54:05.866 INFO:tasks.workunit.client.0.smithi073.stdout:pool 'cache5' is now (or already was) not a tier of 'basepoolB' 2023-11-03T13:54:05.867 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: ceph osd dump 2023-11-03T13:54:05.867 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: grep 'pool.*'\''cache5'\''' 2023-11-03T13:54:05.868 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:565: test_tiering_9: grep 'tier_of[ \t]\+21' 2023-11-03T13:54:06.228 INFO:tasks.workunit.client.0.smithi073.stdout:pool 23 'cache5' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 132 flags hashpspool tier_of 21 stripe_width 0 application rados read_balance_score 1.49 2023-11-03T13:54:06.229 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:566: test_tiering_9: ceph osd tier remove basepoolA cache6 2023-11-03T13:54:06.229 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:566: test_tiering_9: grep 'not a tier of' 2023-11-03T13:54:06.627 INFO:tasks.workunit.client.0.smithi073.stdout:pool 'cache6' is now (or already was) not a tier of 'basepoolA' 2023-11-03T13:54:06.628 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: ceph osd dump 2023-11-03T13:54:06.628 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: grep 'pool.*'\''cache6'\''' 2023-11-03T13:54:06.628 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:567: test_tiering_9: grep 'tier_of[ \t]\+22' 2023-11-03T13:54:06.991 INFO:tasks.workunit.client.0.smithi073.stdout:pool 24 'cache6' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 autoscale_mode off last_change 133 flags hashpspool tier_of 22 stripe_width 0 application rados read_balance_score 2.99 2023-11-03T13:54:06.992 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:569: test_tiering_9: ceph osd tier remove basepoolA cache5 2023-11-03T13:54:06.992 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:569: test_tiering_9: grep 'not a tier of' 2023-11-03T13:54:07.474 INFO:tasks.workunit.client.0.smithi073.stdout:pool 'cache5' is now (or already was) not a tier of 'basepoolA' 2023-11-03T13:54:07.475 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: ceph osd dump 2023-11-03T13:54:07.476 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: grep 'pool.*'\''cache5'\''' 2023-11-03T13:54:07.476 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:570: test_tiering_9: grep tier_of 2023-11-03T13:54:07.842 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:571: test_tiering_9: ceph osd tier remove basepoolB cache6 2023-11-03T13:54:07.842 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:571: test_tiering_9: grep 'not a tier of' 2023-11-03T13:54:08.489 INFO:tasks.workunit.client.0.smithi073.stdout:pool 'cache6' is now (or already was) not a tier of 'basepoolB' 2023-11-03T13:54:08.490 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: ceph osd dump 2023-11-03T13:54:08.490 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: grep 'pool.*'\''cache6'\''' 2023-11-03T13:54:08.490 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:572: test_tiering_9: grep tier_of 2023-11-03T13:54:08.855 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: ceph osd dump 2023-11-03T13:54:08.855 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: grep 'pool.*'\''basepoolA'\''' 2023-11-03T13:54:08.855 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:574: test_tiering_9: grep tiers 2023-11-03T13:54:09.219 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: ceph osd dump 2023-11-03T13:54:09.219 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: grep 'pool.*'\''basepoolB'\''' 2023-11-03T13:54:09.220 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:575: test_tiering_9: grep tiers 2023-11-03T13:54:09.585 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:577: test_tiering_9: ceph osd pool delete cache6 cache6 --yes-i-really-really-mean-it 2023-11-03T13:54:10.475 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache6' does not exist 2023-11-03T13:54:10.485 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:578: test_tiering_9: ceph osd pool delete cache5 cache5 --yes-i-really-really-mean-it 2023-11-03T13:54:11.473 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'cache5' does not exist 2023-11-03T13:54:11.484 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:579: test_tiering_9: ceph osd pool delete basepoolB basepoolB --yes-i-really-really-mean-it 2023-11-03T13:54:12.475 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basepoolB' does not exist 2023-11-03T13:54:12.486 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:580: test_tiering_9: ceph osd pool delete basepoolA basepoolA --yes-i-really-really-mean-it 2023-11-03T13:54:13.486 INFO:tasks.workunit.client.0.smithi073.stderr:pool 'basepoolA' does not exist 2023-11-03T13:54:13.497 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:54:13.864 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_auth 2023-11-03T13:54:13.864 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:585: test_auth: expect_false ceph auth add client.xx mon invalid osd 'allow *' 2023-11-03T13:54:13.865 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:54:13.865 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth add client.xx mon invalid osd 'allow *' 2023-11-03T13:54:14.184 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: mon capability parse failed, stopped at 'invalid' of 'invalid' 2023-11-03T13:54:14.187 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:54:14.188 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:586: test_auth: expect_false ceph auth add client.xx mon 'allow *' osd 'allow *' invalid 'allow *' 2023-11-03T13:54:14.188 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:54:14.188 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth add client.xx mon 'allow *' osd 'allow *' invalid 'allow *' 2023-11-03T13:54:14.510 INFO:tasks.workunit.client.0.smithi073.stderr:Error EINVAL: unknown cap type 'invalid' 2023-11-03T13:54:14.513 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:54:14.514 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:587: test_auth: ceph auth add client.xx mon 'allow *' osd 'allow *' 2023-11-03T13:54:15.010 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:588: test_auth: ceph auth export client.xx 2023-11-03T13:54:15.500 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:589: test_auth: ceph auth add client.xx -i client.xx.keyring 2023-11-03T13:54:15.991 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:590: test_auth: rm -f client.xx.keyring 2023-11-03T13:54:15.992 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:591: test_auth: ceph auth list 2023-11-03T13:54:15.993 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:591: test_auth: grep client.xx 2023-11-03T13:54:16.476 INFO:tasks.workunit.client.0.smithi073.stdout:client.xx 2023-11-03T13:54:16.477 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:592: test_auth: ceph auth ls 2023-11-03T13:54:16.477 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:592: test_auth: grep client.xx 2023-11-03T13:54:16.962 INFO:tasks.workunit.client.0.smithi073.stdout:client.xx 2023-11-03T13:54:16.962 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: ceph auth get client.xx 2023-11-03T13:54:16.963 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: grep caps 2023-11-03T13:54:16.963 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:593: test_auth: grep mon 2023-11-03T13:54:17.450 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow *" 2023-11-03T13:54:17.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: ceph auth get client.xx 2023-11-03T13:54:17.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: grep caps 2023-11-03T13:54:17.452 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:594: test_auth: grep osd 2023-11-03T13:54:17.932 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:17.933 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:595: test_auth: ceph auth get-key client.xx 2023-11-03T13:54:18.422 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:596: test_auth: ceph auth print-key client.xx 2023-11-03T13:54:18.909 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:597: test_auth: ceph auth print_key client.xx 2023-11-03T13:54:19.394 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:598: test_auth: ceph auth caps client.xx osd 'allow rw' 2023-11-03T13:54:19.892 INFO:tasks.workunit.client.0.smithi073.stdout:AQCG+0Rl9H+7MRAA3ckBB4nTlKFV2nPVHRV1lQ==AQCG+0Rl9H+7MRAA3ckBB4nTlKFV2nPVHRV1lQ==AQCG+0Rl9H+7MRAA3ckBB4nTlKFV2nPVHRV1lQ==[client.xx] 2023-11-03T13:54:19.893 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCG+0Rl9H+7MRAA3ckBB4nTlKFV2nPVHRV1lQ== 2023-11-03T13:54:19.893 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow rw" 2023-11-03T13:54:19.893 INFO:tasks.workunit.client.0.smithi073.stderr:updated caps for client.xx 2023-11-03T13:54:19.903 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:599: test_auth: expect_false sh 2023-11-03T13:54:19.904 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:54:19.904 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: sh 2023-11-03T13:54:20.387 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:54:20.388 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: ceph auth get client.xx 2023-11-03T13:54:20.388 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: grep osd 2023-11-03T13:54:20.388 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:600: test_auth: grep 'allow rw' 2023-11-03T13:54:20.874 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow rw" 2023-11-03T13:54:20.875 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:601: test_auth: ceph auth caps client.xx mon 'allow command "osd tree"' 2023-11-03T13:54:21.360 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx] 2023-11-03T13:54:21.360 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCG+0Rl9H+7MRAA3ckBB4nTlKFV2nPVHRV1lQ== 2023-11-03T13:54:21.360 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow command \"osd tree\"" 2023-11-03T13:54:21.361 INFO:tasks.workunit.client.0.smithi073.stderr:updated caps for client.xx 2023-11-03T13:54:21.372 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:602: test_auth: ceph auth export 2023-11-03T13:54:21.373 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:602: test_auth: grep client.xx 2023-11-03T13:54:21.858 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx] 2023-11-03T13:54:21.858 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:603: test_auth: ceph auth export -o authfile 2023-11-03T13:54:22.344 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:604: test_auth: ceph auth import -i authfile 2023-11-03T13:54:22.848 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:606: test_auth: ceph auth export -o authfile2 2023-11-03T13:54:23.346 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:607: test_auth: diff authfile authfile2 2023-11-03T13:54:23.347 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:608: test_auth: rm authfile authfile2 2023-11-03T13:54:23.348 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:609: test_auth: ceph auth del client.xx 2023-11-03T13:54:23.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:610: test_auth: expect_false ceph auth get client.xx 2023-11-03T13:54:23.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:54:23.844 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth get client.xx 2023-11-03T13:54:24.156 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOENT: failed to find client.xx in keyring 2023-11-03T13:54:24.159 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:54:24.160 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth: echo -e 'auth add client.xx mon "allow *" osd "allow *"\n' 2023-11-03T13:54:24.160 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth: ceph 2023-11-03T13:54:24.476 INFO:tasks.workunit.client.0.smithi073.stderr:added key for client.xx 2023-11-03T13:54:24.486 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:614: test_auth: ceph auth get client.xx 2023-11-03T13:54:24.962 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx] 2023-11-03T13:54:24.963 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCQ+0Rl0B0qHBAAGQaD3CWJBOi39uTpOD0AzQ== 2023-11-03T13:54:24.963 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow *" 2023-11-03T13:54:24.963 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:24.974 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:616: test_auth: echo 'auth del client.xx' 2023-11-03T13:54:24.974 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:616: test_auth: ceph 2023-11-03T13:54:25.299 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:617: test_auth: expect_false ceph auth get client.xx 2023-11-03T13:54:25.299 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false: set -x 2023-11-03T13:54:25.300 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: ceph auth get client.xx 2023-11-03T13:54:25.616 INFO:tasks.workunit.client.0.smithi073.stderr:Error ENOENT: failed to find client.xx in keyring 2023-11-03T13:54:25.620 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false: return 0 2023-11-03T13:54:25.621 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:54:25.982 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_auth_profiles 2023-11-03T13:54:25.982 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:622: test_auth_profiles: ceph auth add client.xx-profile-ro mon 'allow profile read-only' mgr 'allow profile read-only' 2023-11-03T13:54:26.479 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:624: test_auth_profiles: ceph auth add client.xx-profile-rw mon 'allow profile read-write' mgr 'allow profile read-write' 2023-11-03T13:54:26.978 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:626: test_auth_profiles: ceph auth add client.xx-profile-rd mon 'allow profile role-definer' 2023-11-03T13:54:27.479 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:628: test_auth_profiles: ceph auth export 2023-11-03T13:54:27.963 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:631: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring status 2023-11-03T13:54:28.443 INFO:tasks.workunit.client.0.smithi073.stdout: cluster: 2023-11-03T13:54:28.443 INFO:tasks.workunit.client.0.smithi073.stdout: id: 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:28.444 INFO:tasks.workunit.client.0.smithi073.stdout: health: HEALTH_OK 2023-11-03T13:54:28.444 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:28.444 INFO:tasks.workunit.client.0.smithi073.stdout: services: 2023-11-03T13:54:28.444 INFO:tasks.workunit.client.0.smithi073.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2023-11-03T13:54:28.444 INFO:tasks.workunit.client.0.smithi073.stdout: mgr: x(active, since 4m) 2023-11-03T13:54:28.445 INFO:tasks.workunit.client.0.smithi073.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2023-11-03T13:54:28.445 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:28.445 INFO:tasks.workunit.client.0.smithi073.stdout: data: 2023-11-03T13:54:28.445 INFO:tasks.workunit.client.0.smithi073.stdout: pools: 2 pools, 9 pgs 2023-11-03T13:54:28.445 INFO:tasks.workunit.client.0.smithi073.stdout: objects: 2 objects, 577 KiB 2023-11-03T13:54:28.446 INFO:tasks.workunit.client.0.smithi073.stdout: usage: 87 MiB used, 270 GiB / 270 GiB avail 2023-11-03T13:54:28.446 INFO:tasks.workunit.client.0.smithi073.stdout: pgs: 9 active+clean 2023-11-03T13:54:28.446 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:28.454 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:632: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring osd dump 2023-11-03T13:54:28.811 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 139 2023-11-03T13:54:28.812 INFO:tasks.workunit.client.0.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:28.812 INFO:tasks.workunit.client.0.smithi073.stdout:created 2023-11-03T13:49:59.452451+0000 2023-11-03T13:54:28.812 INFO:tasks.workunit.client.0.smithi073.stdout:modified 2023-11-03T13:54:13.371713+0000 2023-11-03T13:54:28.812 INFO:tasks.workunit.client.0.smithi073.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-03T13:54:28.813 INFO:tasks.workunit.client.0.smithi073.stdout:crush_version 6 2023-11-03T13:54:28.813 INFO:tasks.workunit.client.0.smithi073.stdout:full_ratio 0.95 2023-11-03T13:54:28.813 INFO:tasks.workunit.client.0.smithi073.stdout:backfillfull_ratio 0.9 2023-11-03T13:54:28.813 INFO:tasks.workunit.client.0.smithi073.stdout:nearfull_ratio 0.85 2023-11-03T13:54:28.814 INFO:tasks.workunit.client.0.smithi073.stdout:require_min_compat_client luminous 2023-11-03T13:54:28.814 INFO:tasks.workunit.client.0.smithi073.stdout:min_compat_client jewel 2023-11-03T13:54:28.814 INFO:tasks.workunit.client.0.smithi073.stdout:require_osd_release reef 2023-11-03T13:54:28.814 INFO:tasks.workunit.client.0.smithi073.stdout:stretch_mode_enabled false 2023-11-03T13:54:28.815 INFO:tasks.workunit.client.0.smithi073.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.99 2023-11-03T13:54:28.815 INFO:tasks.workunit.client.0.smithi073.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 12 flags hashpspool stripe_width 0 application rbd read_balance_score 1.88 2023-11-03T13:54:28.815 INFO:tasks.workunit.client.0.smithi073.stdout:max_osd 3 2023-11-03T13:54:28.815 INFO:tasks.workunit.client.0.smithi073.stdout:osd.0 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6800/1963686469,v1:172.21.15.73:6801/1963686469] [v2:172.21.15.73:6802/1963686469,v1:172.21.15.73:6803/1963686469] exists,up 47768af5-4f35-4087-bb4b-e0a73a13a74b 2023-11-03T13:54:28.816 INFO:tasks.workunit.client.0.smithi073.stdout:osd.1 up in weight 1 up_from 8 up_thru 131 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6808/910736944,v1:172.21.15.73:6809/910736944] [v2:172.21.15.73:6810/910736944,v1:172.21.15.73:6811/910736944] exists,up 8fbd18a9-0046-46d9-aae9-f460f3751db3 2023-11-03T13:54:28.816 INFO:tasks.workunit.client.0.smithi073.stdout:osd.2 up in weight 1 up_from 9 up_thru 130 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6816/1497194653,v1:172.21.15.73:6817/1497194653] [v2:172.21.15.73:6818/1497194653,v1:172.21.15.73:6819/1497194653] exists,up 8e528929-d22b-4938-8bdc-ce337fa128ff 2023-11-03T13:54:28.822 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:633: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring pg dump 2023-11-03T13:54:29.169 INFO:tasks.workunit.client.0.smithi073.stdout:version 267 2023-11-03T13:54:29.170 INFO:tasks.workunit.client.0.smithi073.stdout:stamp 2023-11-03T13:54:27.579960+0000 2023-11-03T13:54:29.170 INFO:tasks.workunit.client.0.smithi073.stdout:last_osdmap_epoch 0 2023-11-03T13:54:29.170 INFO:tasks.workunit.client.0.smithi073.stdout:last_pg_scan 0 2023-11-03T13:54:29.170 INFO:tasks.workunit.client.0.smithi073.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-03T13:54:29.170 INFO:tasks.workunit.client.0.smithi073.stdout:1.0 2 0 0 0 0 590368 0 0 97 0 97 active+clean 2023-11-03T13:50:09.570334+0000 10'97 139:261 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:08.553242+0000 0'0 2023-11-03T13:50:08.553242+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000 0 0 2023-11-03T13:54:29.171 INFO:tasks.workunit.client.0.smithi073.stdout:2.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.570276+0000 0'0 139:133 [1,2] 1 [1,2] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000 0 0 2023-11-03T13:54:29.171 INFO:tasks.workunit.client.0.smithi073.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.569159+0000 0'0 139:138 [2,1] 2 [2,1] 2 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000 0 0 2023-11-03T13:54:29.171 INFO:tasks.workunit.client.0.smithi073.stdout:2.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.568562+0000 0'0 139:138 [2,1] 2 [2,1] 2 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000 0 0 2023-11-03T13:54:29.171 INFO:tasks.workunit.client.0.smithi073.stdout:2.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.570205+0000 0'0 139:132 [0,1] 0 [0,1] 0 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000 0 0 2023-11-03T13:54:29.172 INFO:tasks.workunit.client.0.smithi073.stdout:2.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.572615+0000 0'0 139:133 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000 0 0 2023-11-03T13:54:29.172 INFO:tasks.workunit.client.0.smithi073.stdout:2.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.571603+0000 0'0 139:133 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000 0 0 2023-11-03T13:54:29.172 INFO:tasks.workunit.client.0.smithi073.stdout:2.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.571815+0000 0'0 137:132 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000 0 0 2023-11-03T13:54:29.172 INFO:tasks.workunit.client.0.smithi073.stdout:2.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.572989+0000 0'0 137:132 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000 0 0 2023-11-03T13:54:29.172 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:29.173 INFO:tasks.workunit.client.0.smithi073.stdout:2 0 0 0 0 0 0 0 0 0 0 2023-11-03T13:54:29.173 INFO:tasks.workunit.client.0.smithi073.stdout:1 2 0 0 0 0 590368 0 0 97 97 2023-11-03T13:54:29.173 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:29.173 INFO:tasks.workunit.client.0.smithi073.stdout:sum 2 0 0 0 0 590368 0 0 97 97 2023-11-03T13:54:29.173 INFO:tasks.workunit.client.0.smithi073.stdout:OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 2023-11-03T13:54:29.174 INFO:tasks.workunit.client.0.smithi073.stdout:2 32 MiB 90 GiB 32 MiB 90 GiB [0,1] 3 2 2023-11-03T13:54:29.174 INFO:tasks.workunit.client.0.smithi073.stdout:1 28 MiB 90 GiB 28 MiB 90 GiB [0,2] 9 6 2023-11-03T13:54:29.174 INFO:tasks.workunit.client.0.smithi073.stdout:0 28 MiB 90 GiB 28 MiB 90 GiB [1,2] 6 1 2023-11-03T13:54:29.174 INFO:tasks.workunit.client.0.smithi073.stdout:sum 87 MiB 270 GiB 87 MiB 270 GiB 2023-11-03T13:54:29.174 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:29.175 INFO:tasks.workunit.client.0.smithi073.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-03T13:54:29.175 INFO:tasks.workunit.client.0.smithi073.stderr:dumped all 2023-11-03T13:54:29.180 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:634: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring mon dump 2023-11-03T13:54:29.672 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 1 2023-11-03T13:54:29.672 INFO:tasks.workunit.client.0.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:29.672 INFO:tasks.workunit.client.0.smithi073.stdout:last_changed 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:29.673 INFO:tasks.workunit.client.0.smithi073.stdout:created 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:29.673 INFO:tasks.workunit.client.0.smithi073.stdout:min_mon_release 18 (reef) 2023-11-03T13:54:29.673 INFO:tasks.workunit.client.0.smithi073.stdout:election_strategy: 3 2023-11-03T13:54:29.673 INFO:tasks.workunit.client.0.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2023-11-03T13:54:29.673 INFO:tasks.workunit.client.0.smithi073.stdout:1: [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] mon.b 2023-11-03T13:54:29.674 INFO:tasks.workunit.client.0.smithi073.stdout:2: [v2:172.21.15.73:3302/0,v1:172.21.15.73:6791/0] mon.c 2023-11-03T13:54:29.674 INFO:tasks.workunit.client.0.smithi073.stderr:dumped monmap epoch 1 2023-11-03T13:54:29.683 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:636: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring log foo 2023-11-03T13:54:30.004 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:636: test_auth_profiles: true 2023-11-03T13:54:30.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:637: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:30.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:30.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:30.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:30.006 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:30.006 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:30.006 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:638: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring osd set noout 2023-11-03T13:54:30.270 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:638: test_auth_profiles: true 2023-11-03T13:54:30.271 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:639: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:30.271 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:30.271 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:30.271 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:30.271 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:30.272 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:30.272 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:640: test_auth_profiles: ceph -n client.xx-profile-ro -k client.xx.keyring auth ls 2023-11-03T13:54:30.594 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:640: test_auth_profiles: true 2023-11-03T13:54:30.595 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:641: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:30.595 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:30.595 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:30.595 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:30.596 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:30.596 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:30.596 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:644: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring status 2023-11-03T13:54:31.078 INFO:tasks.workunit.client.0.smithi073.stdout: cluster: 2023-11-03T13:54:31.079 INFO:tasks.workunit.client.0.smithi073.stdout: id: 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:31.079 INFO:tasks.workunit.client.0.smithi073.stdout: health: HEALTH_OK 2023-11-03T13:54:31.079 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.079 INFO:tasks.workunit.client.0.smithi073.stdout: services: 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: mgr: x(active, since 4m) 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: data: 2023-11-03T13:54:31.080 INFO:tasks.workunit.client.0.smithi073.stdout: pools: 2 pools, 9 pgs 2023-11-03T13:54:31.081 INFO:tasks.workunit.client.0.smithi073.stdout: objects: 2 objects, 577 KiB 2023-11-03T13:54:31.081 INFO:tasks.workunit.client.0.smithi073.stdout: usage: 87 MiB used, 270 GiB / 270 GiB avail 2023-11-03T13:54:31.081 INFO:tasks.workunit.client.0.smithi073.stdout: pgs: 9 active+clean 2023-11-03T13:54:31.081 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.090 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:645: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd dump 2023-11-03T13:54:31.450 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 139 2023-11-03T13:54:31.450 INFO:tasks.workunit.client.0.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:31.451 INFO:tasks.workunit.client.0.smithi073.stdout:created 2023-11-03T13:49:59.452451+0000 2023-11-03T13:54:31.451 INFO:tasks.workunit.client.0.smithi073.stdout:modified 2023-11-03T13:54:13.371713+0000 2023-11-03T13:54:31.451 INFO:tasks.workunit.client.0.smithi073.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-03T13:54:31.451 INFO:tasks.workunit.client.0.smithi073.stdout:crush_version 6 2023-11-03T13:54:31.451 INFO:tasks.workunit.client.0.smithi073.stdout:full_ratio 0.95 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:backfillfull_ratio 0.9 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:nearfull_ratio 0.85 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:require_min_compat_client luminous 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:min_compat_client jewel 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:require_osd_release reef 2023-11-03T13:54:31.452 INFO:tasks.workunit.client.0.smithi073.stdout:stretch_mode_enabled false 2023-11-03T13:54:31.453 INFO:tasks.workunit.client.0.smithi073.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 2.99 2023-11-03T13:54:31.453 INFO:tasks.workunit.client.0.smithi073.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 12 flags hashpspool stripe_width 0 application rbd read_balance_score 1.88 2023-11-03T13:54:31.453 INFO:tasks.workunit.client.0.smithi073.stdout:max_osd 3 2023-11-03T13:54:31.453 INFO:tasks.workunit.client.0.smithi073.stdout:osd.0 up in weight 1 up_from 8 up_thru 130 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6800/1963686469,v1:172.21.15.73:6801/1963686469] [v2:172.21.15.73:6802/1963686469,v1:172.21.15.73:6803/1963686469] exists,up 47768af5-4f35-4087-bb4b-e0a73a13a74b 2023-11-03T13:54:31.453 INFO:tasks.workunit.client.0.smithi073.stdout:osd.1 up in weight 1 up_from 8 up_thru 131 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6808/910736944,v1:172.21.15.73:6809/910736944] [v2:172.21.15.73:6810/910736944,v1:172.21.15.73:6811/910736944] exists,up 8fbd18a9-0046-46d9-aae9-f460f3751db3 2023-11-03T13:54:31.454 INFO:tasks.workunit.client.0.smithi073.stdout:osd.2 up in weight 1 up_from 9 up_thru 130 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6816/1497194653,v1:172.21.15.73:6817/1497194653] [v2:172.21.15.73:6818/1497194653,v1:172.21.15.73:6819/1497194653] exists,up 8e528929-d22b-4938-8bdc-ce337fa128ff 2023-11-03T13:54:31.461 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:646: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring pg dump 2023-11-03T13:54:31.812 INFO:tasks.workunit.client.0.smithi073.stdout:version 269 2023-11-03T13:54:31.812 INFO:tasks.workunit.client.0.smithi073.stdout:stamp 2023-11-03T13:54:31.580508+0000 2023-11-03T13:54:31.812 INFO:tasks.workunit.client.0.smithi073.stdout:last_osdmap_epoch 0 2023-11-03T13:54:31.812 INFO:tasks.workunit.client.0.smithi073.stdout:last_pg_scan 0 2023-11-03T13:54:31.813 INFO:tasks.workunit.client.0.smithi073.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-03T13:54:31.813 INFO:tasks.workunit.client.0.smithi073.stdout:1.0 2 0 0 0 0 590368 0 0 97 0 97 active+clean 2023-11-03T13:50:09.570334+0000 10'97 139:261 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:08.553242+0000 0'0 2023-11-03T13:50:08.553242+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000 0 0 2023-11-03T13:54:31.813 INFO:tasks.workunit.client.0.smithi073.stdout:2.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.570276+0000 0'0 139:133 [1,2] 1 [1,2] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000 0 0 2023-11-03T13:54:31.813 INFO:tasks.workunit.client.0.smithi073.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.569159+0000 0'0 139:138 [2,1] 2 [2,1] 2 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000 0 0 2023-11-03T13:54:31.813 INFO:tasks.workunit.client.0.smithi073.stdout:2.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.568562+0000 0'0 139:138 [2,1] 2 [2,1] 2 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000 0 0 2023-11-03T13:54:31.814 INFO:tasks.workunit.client.0.smithi073.stdout:2.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.570205+0000 0'0 139:132 [0,1] 0 [0,1] 0 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000 0 0 2023-11-03T13:54:31.814 INFO:tasks.workunit.client.0.smithi073.stdout:2.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.572615+0000 0'0 139:133 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000 0 0 2023-11-03T13:54:31.814 INFO:tasks.workunit.client.0.smithi073.stdout:2.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.571603+0000 0'0 139:133 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000 0 0 2023-11-03T13:54:31.814 INFO:tasks.workunit.client.0.smithi073.stdout:2.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.571815+0000 0'0 139:133 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000 0 0 2023-11-03T13:54:31.814 INFO:tasks.workunit.client.0.smithi073.stdout:2.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-03T13:50:10.572989+0000 0'0 137:132 [1,0] 1 [1,0] 1 0'0 2023-11-03T13:50:09.557992+0000 0'0 2023-11-03T13:50:09.557992+0000 0 0 periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000 0 0 2023-11-03T13:54:31.815 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.815 INFO:tasks.workunit.client.0.smithi073.stdout:2 0 0 0 0 0 0 0 0 0 0 2023-11-03T13:54:31.815 INFO:tasks.workunit.client.0.smithi073.stdout:1 2 0 0 0 0 590368 0 0 97 97 2023-11-03T13:54:31.815 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.815 INFO:tasks.workunit.client.0.smithi073.stdout:sum 2 0 0 0 0 590368 0 0 97 97 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout:OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout:2 32 MiB 90 GiB 32 MiB 90 GiB [0,1] 3 2 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout:1 28 MiB 90 GiB 28 MiB 90 GiB [0,2] 9 6 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout:0 28 MiB 90 GiB 28 MiB 90 GiB [1,2] 6 1 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout:sum 87 MiB 270 GiB 87 MiB 270 GiB 2023-11-03T13:54:31.816 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:31.817 INFO:tasks.workunit.client.0.smithi073.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-03T13:54:31.817 INFO:tasks.workunit.client.0.smithi073.stderr:dumped all 2023-11-03T13:54:31.822 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:647: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring mon dump 2023-11-03T13:54:32.301 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 1 2023-11-03T13:54:32.301 INFO:tasks.workunit.client.0.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:32.302 INFO:tasks.workunit.client.0.smithi073.stdout:last_changed 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:32.302 INFO:tasks.workunit.client.0.smithi073.stdout:created 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:32.302 INFO:tasks.workunit.client.0.smithi073.stdout:min_mon_release 18 (reef) 2023-11-03T13:54:32.302 INFO:tasks.workunit.client.0.smithi073.stdout:election_strategy: 3 2023-11-03T13:54:32.302 INFO:tasks.workunit.client.0.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2023-11-03T13:54:32.303 INFO:tasks.workunit.client.0.smithi073.stdout:1: [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] mon.b 2023-11-03T13:54:32.303 INFO:tasks.workunit.client.0.smithi073.stdout:2: [v2:172.21.15.73:3302/0,v1:172.21.15.73:6791/0] mon.c 2023-11-03T13:54:32.303 INFO:tasks.workunit.client.0.smithi073.stderr:dumped monmap epoch 1 2023-11-03T13:54:32.312 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:648: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring fs dump 2023-11-03T13:54:32.736 INFO:tasks.workunit.client.0.smithi073.stdout:e1 2023-11-03T13:54:32.737 INFO:tasks.workunit.client.0.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-11-03T13:54:32.737 INFO:tasks.workunit.client.0.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-11-03T13:54:32.737 INFO:tasks.workunit.client.0.smithi073.stdout:legacy client fscid: -1 2023-11-03T13:54:32.738 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:32.738 INFO:tasks.workunit.client.0.smithi073.stdout:No filesystems configured 2023-11-03T13:54:32.738 INFO:tasks.workunit.client.0.smithi073.stderr:dumped fsmap epoch 1 2023-11-03T13:54:32.747 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:649: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring log foo 2023-11-03T13:54:34.466 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:650: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd set noout 2023-11-03T13:54:36.473 INFO:tasks.workunit.client.0.smithi073.stderr:noout is set 2023-11-03T13:54:37.126 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:651: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring osd unset noout 2023-11-03T13:54:38.480 INFO:tasks.workunit.client.0.smithi073.stderr:noout is unset 2023-11-03T13:54:38.494 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:653: test_auth_profiles: ceph -n client.xx-profile-rw -k client.xx.keyring auth ls 2023-11-03T13:54:38.819 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:653: test_auth_profiles: true 2023-11-03T13:54:38.819 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:654: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:38.819 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:38.820 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:38.820 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:38.820 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:38.820 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:38.821 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:657: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth ls 2023-11-03T13:54:39.295 INFO:tasks.workunit.client.0.smithi073.stdout:osd.0 2023-11-03T13:54:39.296 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQBx+kRliD+7NhAALla2c0B9ZilflYQD000YvQ== 2023-11-03T13:54:39.296 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow profile osd 2023-11-03T13:54:39.297 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile osd 2023-11-03T13:54:39.297 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow * 2023-11-03T13:54:39.298 INFO:tasks.workunit.client.0.smithi073.stdout:osd.1 2023-11-03T13:54:39.298 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQB0+kRl4fgaNhAAYb2wZhXJgedkl61JANuB5A== 2023-11-03T13:54:39.298 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow profile osd 2023-11-03T13:54:39.299 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile osd 2023-11-03T13:54:39.299 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow * 2023-11-03T13:54:39.300 INFO:tasks.workunit.client.0.smithi073.stdout:osd.2 2023-11-03T13:54:39.300 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQB3+kRlAaFxNBAA1Q9vokF9aohFGtJuqPoTYg== 2023-11-03T13:54:39.301 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow profile osd 2023-11-03T13:54:39.301 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile osd 2023-11-03T13:54:39.301 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow * 2023-11-03T13:54:39.302 INFO:tasks.workunit.client.0.smithi073.stdout:client.0 2023-11-03T13:54:39.302 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQBv+kRlHD3+FhAA9yh6noUV5w+C3zvkSMhTOw== 2023-11-03T13:54:39.302 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mds] allow 2023-11-03T13:54:39.303 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow r 2023-11-03T13:54:39.303 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow rw 2023-11-03T13:54:39.304 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow rwx 2023-11-03T13:54:39.304 INFO:tasks.workunit.client.0.smithi073.stdout:client.admin 2023-11-03T13:54:39.304 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQBv+kRl9DRFARAAMJgC2Y5tcAydnD6SQaNhWA== 2023-11-03T13:54:39.305 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mds] allow * 2023-11-03T13:54:39.305 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow * 2023-11-03T13:54:39.305 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow * 2023-11-03T13:54:39.306 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow * 2023-11-03T13:54:39.306 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-mds 2023-11-03T13:54:39.307 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRlxQEBGxAALrS4gf1WVWPz+dVqKd7c3g== 2023-11-03T13:54:39.307 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-mds 2023-11-03T13:54:39.307 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-mgr 2023-11-03T13:54:39.308 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRlGwsBGxAA264EdqKSDFrCPnJRWMZ3Wg== 2023-11-03T13:54:39.308 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-mgr 2023-11-03T13:54:39.309 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-osd 2023-11-03T13:54:39.309 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRldRIBGxAAb+fKhyulBmE253V8/YPHLA== 2023-11-03T13:54:39.309 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-osd 2023-11-03T13:54:39.310 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-rbd 2023-11-03T13:54:39.310 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRl7BkBGxAAnyG/FnZX11oSdiTiI9bPmA== 2023-11-03T13:54:39.310 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-rbd 2023-11-03T13:54:39.311 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-rbd-mirror 2023-11-03T13:54:39.312 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRl6SABGxAATRkA5bc/kaDsur4Vjve9OA== 2023-11-03T13:54:39.312 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-rbd-mirror 2023-11-03T13:54:39.312 INFO:tasks.workunit.client.0.smithi073.stdout:client.bootstrap-rgw 2023-11-03T13:54:39.312 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCH+kRlFCgBGxAA0jh7S5/+M9oNrdnLQMcTmQ== 2023-11-03T13:54:39.313 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile bootstrap-rgw 2023-11-03T13:54:39.313 INFO:tasks.workunit.client.0.smithi073.stdout:client.xx-profile-rd 2023-11-03T13:54:39.313 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCT+0RlZv60ERAA2yzUI+Q/1mEvidHl5pyl/g== 2023-11-03T13:54:39.313 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile role-definer 2023-11-03T13:54:39.314 INFO:tasks.workunit.client.0.smithi073.stdout:client.xx-profile-ro 2023-11-03T13:54:39.314 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCS+0Rl6iTKERAAkyRyV5eUHsbR4Ic52PAJbA== 2023-11-03T13:54:39.314 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow profile read-only 2023-11-03T13:54:39.314 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile read-only 2023-11-03T13:54:39.314 INFO:tasks.workunit.client.0.smithi073.stdout:client.xx-profile-rw 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQCS+0RlfC2wLxAA8noeQTmlshG6RxYHxM4Tww== 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mgr] allow profile read-write 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile read-write 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout:mgr.x 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout: key: AQBv+kRlA/YcFBAAKyb88Vdgn8Y6q8KCBIj6yg== 2023-11-03T13:54:39.315 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mds] allow * 2023-11-03T13:54:39.316 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [mon] allow profile mgr 2023-11-03T13:54:39.316 INFO:tasks.workunit.client.0.smithi073.stdout: caps: [osd] allow * 2023-11-03T13:54:39.316 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:658: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth export 2023-11-03T13:54:39.784 INFO:tasks.workunit.client.0.smithi073.stdout:[osd.0] 2023-11-03T13:54:39.785 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQBx+kRliD+7NhAALla2c0B9ZilflYQD000YvQ== 2023-11-03T13:54:39.785 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow profile osd" 2023-11-03T13:54:39.785 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile osd" 2023-11-03T13:54:39.785 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:39.785 INFO:tasks.workunit.client.0.smithi073.stdout:[osd.1] 2023-11-03T13:54:39.786 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQB0+kRl4fgaNhAAYb2wZhXJgedkl61JANuB5A== 2023-11-03T13:54:39.786 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow profile osd" 2023-11-03T13:54:39.786 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile osd" 2023-11-03T13:54:39.786 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:39.786 INFO:tasks.workunit.client.0.smithi073.stdout:[osd.2] 2023-11-03T13:54:39.787 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQB3+kRlAaFxNBAA1Q9vokF9aohFGtJuqPoTYg== 2023-11-03T13:54:39.787 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow profile osd" 2023-11-03T13:54:39.787 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile osd" 2023-11-03T13:54:39.787 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:39.787 INFO:tasks.workunit.client.0.smithi073.stdout:[client.0] 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQBv+kRlHD3+FhAA9yh6noUV5w+C3zvkSMhTOw== 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout: caps mds = "allow" 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow r" 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow rw" 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow rwx" 2023-11-03T13:54:39.788 INFO:tasks.workunit.client.0.smithi073.stdout:[client.admin] 2023-11-03T13:54:39.789 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQBv+kRl9DRFARAAMJgC2Y5tcAydnD6SQaNhWA== 2023-11-03T13:54:39.789 INFO:tasks.workunit.client.0.smithi073.stdout: caps mds = "allow *" 2023-11-03T13:54:39.789 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow *" 2023-11-03T13:54:39.789 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow *" 2023-11-03T13:54:39.789 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:39.790 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-mds] 2023-11-03T13:54:39.790 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRlxQEBGxAALrS4gf1WVWPz+dVqKd7c3g== 2023-11-03T13:54:39.790 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-mds" 2023-11-03T13:54:39.790 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-mgr] 2023-11-03T13:54:39.790 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRlGwsBGxAA264EdqKSDFrCPnJRWMZ3Wg== 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-mgr" 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-osd] 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRldRIBGxAAb+fKhyulBmE253V8/YPHLA== 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-osd" 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-rbd] 2023-11-03T13:54:39.791 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRl7BkBGxAAnyG/FnZX11oSdiTiI9bPmA== 2023-11-03T13:54:39.792 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-rbd" 2023-11-03T13:54:39.792 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-rbd-mirror] 2023-11-03T13:54:39.792 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRl6SABGxAATRkA5bc/kaDsur4Vjve9OA== 2023-11-03T13:54:39.792 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-rbd-mirror" 2023-11-03T13:54:39.792 INFO:tasks.workunit.client.0.smithi073.stdout:[client.bootstrap-rgw] 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCH+kRlFCgBGxAA0jh7S5/+M9oNrdnLQMcTmQ== 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile bootstrap-rgw" 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx-profile-rd] 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCT+0RlZv60ERAA2yzUI+Q/1mEvidHl5pyl/g== 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile role-definer" 2023-11-03T13:54:39.793 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx-profile-ro] 2023-11-03T13:54:39.794 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCS+0Rl6iTKERAAkyRyV5eUHsbR4Ic52PAJbA== 2023-11-03T13:54:39.794 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow profile read-only" 2023-11-03T13:54:39.794 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile read-only" 2023-11-03T13:54:39.794 INFO:tasks.workunit.client.0.smithi073.stdout:[client.xx-profile-rw] 2023-11-03T13:54:39.794 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQCS+0RlfC2wLxAA8noeQTmlshG6RxYHxM4Tww== 2023-11-03T13:54:39.795 INFO:tasks.workunit.client.0.smithi073.stdout: caps mgr = "allow profile read-write" 2023-11-03T13:54:39.795 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile read-write" 2023-11-03T13:54:39.795 INFO:tasks.workunit.client.0.smithi073.stdout:[mgr.x] 2023-11-03T13:54:39.795 INFO:tasks.workunit.client.0.smithi073.stdout: key = AQBv+kRlA/YcFBAAKyb88Vdgn8Y6q8KCBIj6yg== 2023-11-03T13:54:39.795 INFO:tasks.workunit.client.0.smithi073.stdout: caps mds = "allow *" 2023-11-03T13:54:39.796 INFO:tasks.workunit.client.0.smithi073.stdout: caps mon = "allow profile mgr" 2023-11-03T13:54:39.796 INFO:tasks.workunit.client.0.smithi073.stdout: caps osd = "allow *" 2023-11-03T13:54:39.796 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:659: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth add client.xx-profile-foo 2023-11-03T13:54:40.291 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:660: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring status 2023-11-03T13:54:40.772 INFO:tasks.workunit.client.0.smithi073.stdout: cluster: 2023-11-03T13:54:40.773 INFO:tasks.workunit.client.0.smithi073.stdout: id: 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:40.773 INFO:tasks.workunit.client.0.smithi073.stdout: health: HEALTH_OK 2023-11-03T13:54:40.773 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:40.773 INFO:tasks.workunit.client.0.smithi073.stdout: services: 2023-11-03T13:54:40.774 INFO:tasks.workunit.client.0.smithi073.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2023-11-03T13:54:40.774 INFO:tasks.workunit.client.0.smithi073.stdout: mgr: x(active, since 4m) 2023-11-03T13:54:40.774 INFO:tasks.workunit.client.0.smithi073.stdout: osd: 3 osds: 3 up (since 4m), 3 in (since 4m) 2023-11-03T13:54:40.774 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:40.775 INFO:tasks.workunit.client.0.smithi073.stdout: data: 2023-11-03T13:54:40.775 INFO:tasks.workunit.client.0.smithi073.stdout: pools: 2 pools, 9 pgs 2023-11-03T13:54:40.775 INFO:tasks.workunit.client.0.smithi073.stdout: objects: 2 objects, 577 KiB 2023-11-03T13:54:40.775 INFO:tasks.workunit.client.0.smithi073.stdout: usage: 87 MiB used, 270 GiB / 270 GiB avail 2023-11-03T13:54:40.775 INFO:tasks.workunit.client.0.smithi073.stdout: pgs: 9 active+clean 2023-11-03T13:54:40.776 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:40.783 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:661: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring osd dump 2023-11-03T13:54:41.040 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:661: test_auth_profiles: true 2023-11-03T13:54:41.041 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:662: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:41.041 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:41.041 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:41.042 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:41.042 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:41.042 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:41.043 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:663: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring pg dump 2023-11-03T13:54:41.324 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-11-03T13:54:41.322+0000 7fb02ab3d640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2023-11-03T13:54:41.331 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:663: test_auth_profiles: true 2023-11-03T13:54:41.331 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:664: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:41.332 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:41.332 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:41.332 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:41.333 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:41.333 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:41.334 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:666: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring mon dump 2023-11-03T13:54:41.822 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 1 2023-11-03T13:54:41.822 INFO:tasks.workunit.client.0.smithi073.stdout:fsid 3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:41.822 INFO:tasks.workunit.client.0.smithi073.stdout:last_changed 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:41.823 INFO:tasks.workunit.client.0.smithi073.stdout:created 2023-11-03T13:49:34.742002+0000 2023-11-03T13:54:41.823 INFO:tasks.workunit.client.0.smithi073.stdout:min_mon_release 18 (reef) 2023-11-03T13:54:41.823 INFO:tasks.workunit.client.0.smithi073.stdout:election_strategy: 3 2023-11-03T13:54:41.823 INFO:tasks.workunit.client.0.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2023-11-03T13:54:41.824 INFO:tasks.workunit.client.0.smithi073.stdout:1: [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] mon.b 2023-11-03T13:54:41.824 INFO:tasks.workunit.client.0.smithi073.stdout:2: [v2:172.21.15.73:3302/0,v1:172.21.15.73:6791/0] mon.c 2023-11-03T13:54:41.824 INFO:tasks.workunit.client.0.smithi073.stderr:dumped monmap epoch 1 2023-11-03T13:54:41.833 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:668: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring mon add foo 1.1.1.1 2023-11-03T13:54:42.157 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:668: test_auth_profiles: true 2023-11-03T13:54:42.157 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:669: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:42.157 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:42.158 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:42.158 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:42.158 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:42.158 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:42.159 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:670: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring fs dump 2023-11-03T13:54:42.449 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:670: test_auth_profiles: true 2023-11-03T13:54:42.449 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:671: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:42.450 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:42.450 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:42.450 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:42.450 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:42.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:42.451 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:672: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring log foo 2023-11-03T13:54:42.775 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:672: test_auth_profiles: true 2023-11-03T13:54:42.775 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:673: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:42.775 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:42.776 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:42.776 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:42.776 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:42.776 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:42.777 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:674: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring osd set noout 2023-11-03T13:54:43.044 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:674: test_auth_profiles: true 2023-11-03T13:54:43.044 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:675: test_auth_profiles: check_response 'EACCES: access denied' 2023-11-03T13:54:43.044 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:103: check_response: expected_string='EACCES: access denied' 2023-11-03T13:54:43.044 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:104: check_response: retcode= 2023-11-03T13:54:43.045 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:105: check_response: expected_retcode= 2023-11-03T13:54:43.045 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:106: check_response: '[' '' -a '!=' ']' 2023-11-03T13:54:43.045 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:111: check_response: grep --quiet -- 'EACCES: access denied' /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:43.045 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:677: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth del client.xx-profile-ro 2023-11-03T13:54:43.542 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:678: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth del client.xx-profile-rw 2023-11-03T13:54:44.032 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:681: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth add client.xx-profile-rd2 mon 'allow profile role-definer' 2023-11-03T13:54:44.526 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:683: test_auth_profiles: ceph -n client.xx-profile-rd -k client.xx.keyring auth export 2023-11-03T13:54:45.013 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:686: test_auth_profiles: ceph -n client.xx-profile-rd2 -k client.xx.keyring.2 auth del client.xx-profile-rd 2023-11-03T13:54:45.510 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:689: test_auth_profiles: ceph auth del client.xx-profile-rd2 2023-11-03T13:54:46.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:690: test_auth_profiles: rm -f client.xx.keyring client.xx.keyring.2 2023-11-03T13:54:46.005 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2983: : set +x 2023-11-03T13:54:46.366 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2982: : test_mon_misc 2023-11-03T13:54:46.366 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:719: test_mon_misc: ceph osd dump 2023-11-03T13:54:46.367 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:719: test_mon_misc: grep '^epoch' 2023-11-03T13:54:46.730 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 143 2023-11-03T13:54:46.731 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:720: test_mon_misc: ceph --concise osd dump 2023-11-03T13:54:46.731 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:720: test_mon_misc: grep '^epoch' 2023-11-03T13:54:47.093 INFO:tasks.workunit.client.0.smithi073.stdout:epoch 143 2023-11-03T13:54:47.094 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:722: test_mon_misc: ceph osd df 2023-11-03T13:54:47.094 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:722: test_mon_misc: grep 'MIN/MAX VAR' 2023-11-03T13:54:47.455 INFO:tasks.workunit.client.0.smithi073.stdout:MIN/MAX VAR: 0.95/1.09 STDDEV: 0.00 2023-11-03T13:54:47.455 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:725: test_mon_misc: ceph df 2023-11-03T13:54:47.945 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:726: test_mon_misc: grep RAW /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:47.946 INFO:tasks.workunit.client.0.smithi073.stdout:--- RAW STORAGE --- 2023-11-03T13:54:47.946 INFO:tasks.workunit.client.0.smithi073.stdout:CLASS SIZE AVAIL USED RAW USED %RAW USED 2023-11-03T13:54:47.947 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:727: test_mon_misc: grep -v DIRTY /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:47.947 INFO:tasks.workunit.client.0.smithi073.stdout:--- RAW STORAGE --- 2023-11-03T13:54:47.947 INFO:tasks.workunit.client.0.smithi073.stdout:CLASS SIZE AVAIL USED RAW USED %RAW USED 2023-11-03T13:54:47.948 INFO:tasks.workunit.client.0.smithi073.stdout:ssd 270 GiB 270 GiB 87 MiB 87 MiB 0.03 2023-11-03T13:54:47.948 INFO:tasks.workunit.client.0.smithi073.stdout:TOTAL 270 GiB 270 GiB 87 MiB 87 MiB 0.03 2023-11-03T13:54:47.948 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:47.948 INFO:tasks.workunit.client.0.smithi073.stdout:--- POOLS --- 2023-11-03T13:54:47.948 INFO:tasks.workunit.client.0.smithi073.stdout:POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL 2023-11-03T13:54:47.949 INFO:tasks.workunit.client.0.smithi073.stdout:.mgr 1 1 456 KiB 2 104 KiB 0 128 GiB 2023-11-03T13:54:47.949 INFO:tasks.workunit.client.0.smithi073.stdout:rbd 2 8 0 B 0 0 B 0 128 GiB 2023-11-03T13:54:47.949 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:728: test_mon_misc: ceph df detail 2023-11-03T13:54:48.438 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:729: test_mon_misc: grep DIRTY /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:48.439 INFO:tasks.workunit.client.0.smithi073.stdout:POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR 2023-11-03T13:54:48.439 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:730: test_mon_misc: ceph df --format json 2023-11-03T13:54:48.930 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:731: test_mon_misc: grep total_bytes /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:48.931 INFO:tasks.workunit.client.0.smithi073.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661}},"pools":[{"name":".mgr","id":1,"stats":{"stored":466956,"objects":2,"kb_used":104,"bytes_used":106496,"percent_used":3.8681537262164056e-07,"max_avail":137657352192}},{"name":"rbd","id":2,"stats":{"stored":0,"objects":0,"kb_used":0,"bytes_used":0,"percent_used":0,"max_avail":137657352192}}]} 2023-11-03T13:54:48.932 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:732: test_mon_misc: grep -v dirty /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:48.932 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:48.932 INFO:tasks.workunit.client.0.smithi073.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661}},"pools":[{"name":".mgr","id":1,"stats":{"stored":466956,"objects":2,"kb_used":104,"bytes_used":106496,"percent_used":3.8681537262164056e-07,"max_avail":137657352192}},{"name":"rbd","id":2,"stats":{"stored":0,"objects":0,"kb_used":0,"bytes_used":0,"percent_used":0,"max_avail":137657352192}}]} 2023-11-03T13:54:48.933 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:733: test_mon_misc: ceph df detail --format json 2023-11-03T13:54:49.433 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:734: test_mon_misc: grep rd_bytes /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:49.433 INFO:tasks.workunit.client.0.smithi073.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661}},"pools":[{"name":".mgr","id":1,"stats":{"stored":466956,"stored_data":466956,"stored_omap":0,"objects":2,"kb_used":104,"bytes_used":106496,"data_bytes_used":106496,"omap_bytes_used":0,"percent_used":3.8681537262164056e-07,"max_avail":137657352192,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":106,"rd_bytes":93184,"wr":137,"wr_bytes":1939456,"compress_bytes_used":65536,"compress_under_bytes":901120,"stored_raw":933912,"avail_raw":275314692722}},{"name":"rbd","id":2,"stats":{"stored":0,"stored_data":0,"stored_omap":0,"objects":0,"kb_used":0,"bytes_used":0,"data_bytes_used":0,"omap_bytes_used":0,"percent_used":0,"max_avail":137657352192,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":0,"rd_bytes":0,"wr":0,"wr_bytes":0,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":0,"avail_raw":275314692722}}]} 2023-11-03T13:54:49.434 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:735: test_mon_misc: grep dirty /tmp/cephtool.KY8/test_invalid.12p 2023-11-03T13:54:49.434 INFO:tasks.workunit.client.0.smithi073.stdout:{"stats":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3},"stats_by_class":{"ssd":{"total_bytes":289910292480,"total_avail_bytes":289818640384,"total_used_bytes":91652096,"total_used_raw_bytes":91652096,"total_used_raw_ratio":0.00031613951432518661}},"pools":[{"name":".mgr","id":1,"stats":{"stored":466956,"stored_data":466956,"stored_omap":0,"objects":2,"kb_used":104,"bytes_used":106496,"data_bytes_used":106496,"omap_bytes_used":0,"percent_used":3.8681537262164056e-07,"max_avail":137657352192,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":106,"rd_bytes":93184,"wr":137,"wr_bytes":1939456,"compress_bytes_used":65536,"compress_under_bytes":901120,"stored_raw":933912,"avail_raw":275314692722}},{"name":"rbd","id":2,"stats":{"stored":0,"stored_data":0,"stored_omap":0,"objects":0,"kb_used":0,"bytes_used":0,"data_bytes_used":0,"omap_bytes_used":0,"percent_used":0,"max_avail":137657352192,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":0,"rd_bytes":0,"wr":0,"wr_bytes":0,"compress_bytes_used":0,"compress_under_bytes":0,"stored_raw":0,"avail_raw":275314692722}}]} 2023-11-03T13:54:49.435 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:736: test_mon_misc: ceph df --format xml 2023-11-03T13:54:49.435 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:736: test_mon_misc: grep '' 2023-11-03T13:54:49.926 INFO:tasks.workunit.client.0.smithi073.stdout:28991029248028981864038491652096916520960.0003161395143251866133328991029248028981864038491652096916520960.00031613951432518661.mgr146695621041064963.8681537262164056e-07137657352192rbd200000137657352192 2023-11-03T13:54:49.927 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:737: test_mon_misc: ceph df detail --format xml 2023-11-03T13:54:49.927 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:737: test_mon_misc: grep '' 2023-11-03T13:54:50.421 INFO:tasks.workunit.client.0.smithi073.stdout:28991029248028981864038491652096916520960.0003161395143251866133328991029248028981864038491652096916520960.00031613951432518661.mgr14669564669560210410649610649603.8681537262164056e-0713765735219200010693184137193945665536901120933912275314692722rbd20000000001376573521920000000000275314692722 2023-11-03T13:54:50.422 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:739: test_mon_misc: ceph fsid 2023-11-03T13:54:50.900 INFO:tasks.workunit.client.0.smithi073.stdout:3628883d-b18e-4d59-aee7-61c0fc38535d 2023-11-03T13:54:50.911 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:740: test_mon_misc: ceph health 2023-11-03T13:54:51.331 INFO:tasks.workunit.client.0.smithi073.stdout:HEALTH_OK 2023-11-03T13:54:51.341 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:741: test_mon_misc: ceph health detail 2023-11-03T13:54:51.824 INFO:tasks.workunit.client.0.smithi073.stdout:HEALTH_OK 2023-11-03T13:54:51.834 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:742: test_mon_misc: ceph health --format json-pretty 2023-11-03T13:54:52.252 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.253 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:52.253 INFO:tasks.workunit.client.0.smithi073.stdout: "status": "HEALTH_OK", 2023-11-03T13:54:52.253 INFO:tasks.workunit.client.0.smithi073.stdout: "checks": {}, 2023-11-03T13:54:52.254 INFO:tasks.workunit.client.0.smithi073.stdout: "mutes": [] 2023-11-03T13:54:52.254 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:52.264 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:743: test_mon_misc: ceph health detail --format xml-pretty 2023-11-03T13:54:52.741 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.742 INFO:tasks.workunit.client.0.smithi073.stdout: HEALTH_OK 2023-11-03T13:54:52.742 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.742 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.742 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.742 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.743 INFO:tasks.workunit.client.0.smithi073.stdout: 2023-11-03T13:54:52.752 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:745: test_mon_misc: ceph time-sync-status 2023-11-03T13:54:53.226 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:53.226 INFO:tasks.workunit.client.0.smithi073.stdout: "time_skew_status": { 2023-11-03T13:54:53.226 INFO:tasks.workunit.client.0.smithi073.stdout: "a": { 2023-11-03T13:54:53.226 INFO:tasks.workunit.client.0.smithi073.stdout: "skew": 0, 2023-11-03T13:54:53.227 INFO:tasks.workunit.client.0.smithi073.stdout: "latency": 0, 2023-11-03T13:54:53.227 INFO:tasks.workunit.client.0.smithi073.stdout: "health": "HEALTH_OK" 2023-11-03T13:54:53.227 INFO:tasks.workunit.client.0.smithi073.stdout: }, 2023-11-03T13:54:53.227 INFO:tasks.workunit.client.0.smithi073.stdout: "b": { 2023-11-03T13:54:53.227 INFO:tasks.workunit.client.0.smithi073.stdout: "skew": 0, 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: "latency": 0.000866846, 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: "health": "HEALTH_OK" 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: }, 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: "c": { 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: "skew": 0, 2023-11-03T13:54:53.228 INFO:tasks.workunit.client.0.smithi073.stdout: "latency": 0.0012163269999999999, 2023-11-03T13:54:53.229 INFO:tasks.workunit.client.0.smithi073.stdout: "health": "HEALTH_OK" 2023-11-03T13:54:53.229 INFO:tasks.workunit.client.0.smithi073.stdout: } 2023-11-03T13:54:53.229 INFO:tasks.workunit.client.0.smithi073.stdout: }, 2023-11-03T13:54:53.229 INFO:tasks.workunit.client.0.smithi073.stdout: "timechecks": { 2023-11-03T13:54:53.229 INFO:tasks.workunit.client.0.smithi073.stdout: "epoch": 10, 2023-11-03T13:54:53.230 INFO:tasks.workunit.client.0.smithi073.stdout: "round": 2, 2023-11-03T13:54:53.230 INFO:tasks.workunit.client.0.smithi073.stdout: "round_status": "finished" 2023-11-03T13:54:53.230 INFO:tasks.workunit.client.0.smithi073.stdout: } 2023-11-03T13:54:53.230 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:53.236 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:747: test_mon_misc: ceph node ls 2023-11-03T13:54:53.721 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:53.721 INFO:tasks.workunit.client.0.smithi073.stdout: "mon": { 2023-11-03T13:54:53.721 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:53.721 INFO:tasks.workunit.client.0.smithi073.stdout: "a", 2023-11-03T13:54:53.722 INFO:tasks.workunit.client.0.smithi073.stdout: "b", 2023-11-03T13:54:53.722 INFO:tasks.workunit.client.0.smithi073.stdout: "c" 2023-11-03T13:54:53.722 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:53.722 INFO:tasks.workunit.client.0.smithi073.stdout: }, 2023-11-03T13:54:53.722 INFO:tasks.workunit.client.0.smithi073.stdout: "osd": { 2023-11-03T13:54:53.723 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:53.723 INFO:tasks.workunit.client.0.smithi073.stdout: 0, 2023-11-03T13:54:53.723 INFO:tasks.workunit.client.0.smithi073.stdout: 1, 2023-11-03T13:54:53.723 INFO:tasks.workunit.client.0.smithi073.stdout: 2 2023-11-03T13:54:53.723 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: }, 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: "mgr": { 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: "x" 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:53.724 INFO:tasks.workunit.client.0.smithi073.stdout: } 2023-11-03T13:54:53.725 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:53.732 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2023-11-03T13:54:53.732 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mon 2023-11-03T13:54:54.214 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:54.214 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:54.214 INFO:tasks.workunit.client.0.smithi073.stdout: "a", 2023-11-03T13:54:54.215 INFO:tasks.workunit.client.0.smithi073.stdout: "b", 2023-11-03T13:54:54.215 INFO:tasks.workunit.client.0.smithi073.stdout: "c" 2023-11-03T13:54:54.215 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:54.215 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:54.224 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2023-11-03T13:54:54.224 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls osd 2023-11-03T13:54:54.708 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:54.708 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:54.709 INFO:tasks.workunit.client.0.smithi073.stdout: 0, 2023-11-03T13:54:54.709 INFO:tasks.workunit.client.0.smithi073.stdout: 1, 2023-11-03T13:54:54.709 INFO:tasks.workunit.client.0.smithi073.stdout: 2 2023-11-03T13:54:54.709 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:54.709 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:54.719 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2023-11-03T13:54:54.719 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mds 2023-11-03T13:54:55.211 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:748: test_mon_misc: for t in mon osd mds mgr 2023-11-03T13:54:55.211 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:749: test_mon_misc: ceph node ls mgr 2023-11-03T13:54:55.697 INFO:tasks.workunit.client.0.smithi073.stdout:{ 2023-11-03T13:54:55.697 INFO:tasks.workunit.client.0.smithi073.stdout: "smithi073": [ 2023-11-03T13:54:55.697 INFO:tasks.workunit.client.0.smithi073.stdout: "x" 2023-11-03T13:54:55.697 INFO:tasks.workunit.client.0.smithi073.stdout: ] 2023-11-03T13:54:55.698 INFO:tasks.workunit.client.0.smithi073.stdout:} 2023-11-03T13:54:55.708 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:752: test_mon_misc: ceph_watch_start 2023-11-03T13:54:55.708 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:155: ceph_watch_start: local whatch_opt=--watch 2023-11-03T13:54:55.708 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:157: ceph_watch_start: '[' -n '' ']' 2023-11-03T13:54:55.709 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:164: ceph_watch_start: CEPH_WATCH_FILE=/tmp/cephtool.KY8/CEPH_WATCH_88440 2023-11-03T13:54:55.709 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:166: ceph_watch_start: CEPH_WATCH_PID=96294 2023-11-03T13:54:55.709 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:165: ceph_watch_start: ceph --watch 2023-11-03T13:54:55.709 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: seq 3 2023-11-03T13:54:55.710 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2023-11-03T13:54:55.710 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.KY8/CEPH_WATCH_88440 2023-11-03T13:54:55.710 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2023-11-03T13:54:56.711 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2023-11-03T13:54:56.711 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.KY8/CEPH_WATCH_88440 2023-11-03T13:54:56.712 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2023-11-03T13:54:57.714 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:170: ceph_watch_start: for i in `seq 3` 2023-11-03T13:54:57.714 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:171: ceph_watch_start: grep -q cluster /tmp/cephtool.KY8/CEPH_WATCH_88440 2023-11-03T13:54:57.715 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:172: ceph_watch_start: sleep 1 2023-11-03T13:54:58.717 INFO:tasks.workunit.client.0.smithi073.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:753: test_mon_misc: date 2023-11-03T13:54:58.718 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:753: test_mon_misc: mymsg='this is a test log message 88440.Fri Nov 3 01:54:58 PM UTC 2023' 2023-11-03T13:54:58.718 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:754: test_mon_misc: ceph log 'this is a test log message 88440.Fri Nov 3 01:54:58 PM UTC 2023' 2023-11-03T13:55:00.225 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:755: test_mon_misc: ceph log last 2023-11-03T13:55:00.225 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:755: test_mon_misc: grep 'this is a test log message 88440.Fri Nov 3 01:54:58 PM UTC 2023' 2023-11-03T13:55:00.716 INFO:tasks.workunit.client.0.smithi073.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_mon_misc: rm -fr /tmp/cephtool.KY8 2023-11-03T13:55:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:56:00.718 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-11-03T13:57:00.720 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-11-03T13:57:00.721 INFO:tasks.workunit:Stopping ['cephtool'] on client.0... 2023-11-03T13:57:00.721 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-03T13:57:00.987 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=10145169af62d2c41c01dd8795eeddb1b869c21a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2023-11-03T13:57:01.057 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6475c9c06d9f463bafdf41f2a91ce6ab Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_10145169af62d2c41c01dd8795eeddb1b869c21a/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=10145169af62d2c41c01dd8795eeddb1b869c21a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2023-11-03T13:57:01.061 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-03T13:57:01.070 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-03T13:57:01.071 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:57:01.359 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:01.359 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:57:01.372 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2023-11-03T13:56:59.609007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89504,"kb_used_data":4268,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283026016,"statfs":{"total":289910292480,"available":289818640384,"internally_reserved":0,"allocated":4370432,"data_stored":2741808,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002188"},"pg_stats":[{"pgid":"1.0","version":"10'97","reported_seq":266,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741248+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:54:39.741248+0000","last_peered":"2023-11-03T13:54:39.741248+0000","last_clean":"2023-11-03T13:54:39.741248+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:54:39.741248+0000","last_undegraded":"2023-11-03T13:54:39.741248+0000","last_fullsized":"2023-11-03T13:54:39.741248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741246+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:54:39.741246+0000","last_peered":"2023-11-03T13:54:39.741246+0000","last_clean":"2023-11-03T13:54:39.741246+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:54:39.741246+0000","last_undegraded":"2023-11-03T13:54:39.741246+0000","last_fullsized":"2023-11-03T13:54:39.741246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478412+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:54:38.478412+0000","last_peered":"2023-11-03T13:54:38.478412+0000","last_clean":"2023-11-03T13:54:38.478412+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:54:38.478412+0000","last_undegraded":"2023-11-03T13:54:38.478412+0000","last_fullsized":"2023-11-03T13:54:38.478412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478399+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:54:38.478399+0000","last_peered":"2023-11-03T13:54:38.478399+0000","last_clean":"2023-11-03T13:54:38.478399+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:54:38.478399+0000","last_undegraded":"2023-11-03T13:54:38.478399+0000","last_fullsized":"2023-11-03T13:54:38.478399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478406+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:54:38.478406+0000","last_peered":"2023-11-03T13:54:38.478406+0000","last_clean":"2023-11-03T13:54:38.478406+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:54:38.478406+0000","last_undegraded":"2023-11-03T13:54:38.478406+0000","last_fullsized":"2023-11-03T13:54:38.478406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741271+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:54:39.741271+0000","last_peered":"2023-11-03T13:54:39.741271+0000","last_clean":"2023-11-03T13:54:39.741271+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:54:39.741271+0000","last_undegraded":"2023-11-03T13:54:39.741271+0000","last_fullsized":"2023-11-03T13:54:39.741271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741298+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:54:39.741298+0000","last_peered":"2023-11-03T13:54:39.741298+0000","last_clean":"2023-11-03T13:54:39.741298+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:54:39.741298+0000","last_undegraded":"2023-11-03T13:54:39.741298+0000","last_fullsized":"2023-11-03T13:54:39.741298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741308+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:54:39.741308+0000","last_peered":"2023-11-03T13:54:39.741308+0000","last_clean":"2023-11-03T13:54:39.741308+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:54:39.741308+0000","last_undegraded":"2023-11-03T13:54:39.741308+0000","last_fullsized":"2023-11-03T13:54:39.741308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741312+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:54:39.741312+0000","last_peered":"2023-11-03T13:54:39.741312+0000","last_clean":"2023-11-03T13:54:39.741312+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:54:39.741312+0000","last_undegraded":"2023-11-03T13:54:39.741312+0000","last_fullsized":"2023-11-03T13:54:39.741312+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":602632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.32000000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45100000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.29599999999999999,"15min":0.32200000000000001},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46800000000000003,"5min":0.5,"15min":0.5},"last":0.497}]},{"osd":1,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.31900000000000001},"min":{"1min":0.26600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41499999999999998,"5min":0.45800000000000002,"15min":0.53900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.55100000000000005},"last":0.47999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738461,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.30299999999999999},"min":{"1min":0.23400000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.307,"15min":0.32700000000000001},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.498,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36199999999999999}]},{"osd":2,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.29699999999999999,"15min":0.33700000000000002},"min":{"1min":0.26800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.318},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.33900000000000002},"min":{"1min":0.27300000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343372,"statfs":{"total":96636764160,"available":96607612928,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.29299999999999998,"15min":0.309},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46899999999999997,"5min":0.505,"15min":0.505},"last":0.246},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.308,"15min":0.316},"min":{"1min":0.27500000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.29599999999999999,"15min":0.318},"min":{"1min":0.24199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.27000000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:57:01.373 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:57:01.659 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:01.660 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:57:01.672 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2023-11-03T13:57:01.609290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89504,"kb_used_data":4268,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283026016,"statfs":{"total":289910292480,"available":289818640384,"internally_reserved":0,"allocated":4370432,"data_stored":2741808,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002090"},"pg_stats":[{"pgid":"1.0","version":"10'97","reported_seq":266,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741248+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:54:39.741248+0000","last_peered":"2023-11-03T13:54:39.741248+0000","last_clean":"2023-11-03T13:54:39.741248+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:54:39.741248+0000","last_undegraded":"2023-11-03T13:54:39.741248+0000","last_fullsized":"2023-11-03T13:54:39.741248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741246+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:54:39.741246+0000","last_peered":"2023-11-03T13:54:39.741246+0000","last_clean":"2023-11-03T13:54:39.741246+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:54:39.741246+0000","last_undegraded":"2023-11-03T13:54:39.741246+0000","last_fullsized":"2023-11-03T13:54:39.741246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478412+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:54:38.478412+0000","last_peered":"2023-11-03T13:54:38.478412+0000","last_clean":"2023-11-03T13:54:38.478412+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:54:38.478412+0000","last_undegraded":"2023-11-03T13:54:38.478412+0000","last_fullsized":"2023-11-03T13:54:38.478412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478399+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:54:38.478399+0000","last_peered":"2023-11-03T13:54:38.478399+0000","last_clean":"2023-11-03T13:54:38.478399+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:54:38.478399+0000","last_undegraded":"2023-11-03T13:54:38.478399+0000","last_fullsized":"2023-11-03T13:54:38.478399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478406+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:54:38.478406+0000","last_peered":"2023-11-03T13:54:38.478406+0000","last_clean":"2023-11-03T13:54:38.478406+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:54:38.478406+0000","last_undegraded":"2023-11-03T13:54:38.478406+0000","last_fullsized":"2023-11-03T13:54:38.478406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741271+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:54:39.741271+0000","last_peered":"2023-11-03T13:54:39.741271+0000","last_clean":"2023-11-03T13:54:39.741271+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:54:39.741271+0000","last_undegraded":"2023-11-03T13:54:39.741271+0000","last_fullsized":"2023-11-03T13:54:39.741271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741298+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:54:39.741298+0000","last_peered":"2023-11-03T13:54:39.741298+0000","last_clean":"2023-11-03T13:54:39.741298+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:54:39.741298+0000","last_undegraded":"2023-11-03T13:54:39.741298+0000","last_fullsized":"2023-11-03T13:54:39.741298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741308+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:54:39.741308+0000","last_peered":"2023-11-03T13:54:39.741308+0000","last_clean":"2023-11-03T13:54:39.741308+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:54:39.741308+0000","last_undegraded":"2023-11-03T13:54:39.741308+0000","last_fullsized":"2023-11-03T13:54:39.741308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741312+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:54:39.741312+0000","last_peered":"2023-11-03T13:54:39.741312+0000","last_clean":"2023-11-03T13:54:39.741312+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:54:39.741312+0000","last_undegraded":"2023-11-03T13:54:39.741312+0000","last_fullsized":"2023-11-03T13:54:39.741312+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":602632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.32000000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45100000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.29599999999999999,"15min":0.32200000000000001},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46800000000000003,"5min":0.5,"15min":0.5},"last":0.497}]},{"osd":1,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.31900000000000001},"min":{"1min":0.26600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41499999999999998,"5min":0.45800000000000002,"15min":0.53900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.55100000000000005},"last":0.47999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738461,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.30299999999999999},"min":{"1min":0.23400000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.307,"15min":0.32700000000000001},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.498,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36199999999999999}]},{"osd":2,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.29699999999999999,"15min":0.33700000000000002},"min":{"1min":0.26800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.318},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.33900000000000002},"min":{"1min":0.27300000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343372,"statfs":{"total":96636764160,"available":96607612928,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.29299999999999998,"15min":0.309},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46899999999999997,"5min":0.505,"15min":0.505},"last":0.246},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.308,"15min":0.316},"min":{"1min":0.27500000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.29599999999999999,"15min":0.318},"min":{"1min":0.24199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.27000000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:57:01.673 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-03T13:57:01.674 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:57:01.960 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:01.961 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:57:01.971 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2023-11-03T13:57:01.609290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89504,"kb_used_data":4268,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283026016,"statfs":{"total":289910292480,"available":289818640384,"internally_reserved":0,"allocated":4370432,"data_stored":2741808,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002090"},"pg_stats":[{"pgid":"1.0","version":"10'97","reported_seq":266,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741248+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:54:39.741248+0000","last_peered":"2023-11-03T13:54:39.741248+0000","last_clean":"2023-11-03T13:54:39.741248+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:54:39.741248+0000","last_undegraded":"2023-11-03T13:54:39.741248+0000","last_fullsized":"2023-11-03T13:54:39.741248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741246+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:54:39.741246+0000","last_peered":"2023-11-03T13:54:39.741246+0000","last_clean":"2023-11-03T13:54:39.741246+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:54:39.741246+0000","last_undegraded":"2023-11-03T13:54:39.741246+0000","last_fullsized":"2023-11-03T13:54:39.741246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478412+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:54:38.478412+0000","last_peered":"2023-11-03T13:54:38.478412+0000","last_clean":"2023-11-03T13:54:38.478412+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:54:38.478412+0000","last_undegraded":"2023-11-03T13:54:38.478412+0000","last_fullsized":"2023-11-03T13:54:38.478412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478399+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:54:38.478399+0000","last_peered":"2023-11-03T13:54:38.478399+0000","last_clean":"2023-11-03T13:54:38.478399+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:54:38.478399+0000","last_undegraded":"2023-11-03T13:54:38.478399+0000","last_fullsized":"2023-11-03T13:54:38.478399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478406+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:54:38.478406+0000","last_peered":"2023-11-03T13:54:38.478406+0000","last_clean":"2023-11-03T13:54:38.478406+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:54:38.478406+0000","last_undegraded":"2023-11-03T13:54:38.478406+0000","last_fullsized":"2023-11-03T13:54:38.478406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741271+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:54:39.741271+0000","last_peered":"2023-11-03T13:54:39.741271+0000","last_clean":"2023-11-03T13:54:39.741271+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:54:39.741271+0000","last_undegraded":"2023-11-03T13:54:39.741271+0000","last_fullsized":"2023-11-03T13:54:39.741271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741298+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:54:39.741298+0000","last_peered":"2023-11-03T13:54:39.741298+0000","last_clean":"2023-11-03T13:54:39.741298+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:54:39.741298+0000","last_undegraded":"2023-11-03T13:54:39.741298+0000","last_fullsized":"2023-11-03T13:54:39.741298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741308+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:54:39.741308+0000","last_peered":"2023-11-03T13:54:39.741308+0000","last_clean":"2023-11-03T13:54:39.741308+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:54:39.741308+0000","last_undegraded":"2023-11-03T13:54:39.741308+0000","last_fullsized":"2023-11-03T13:54:39.741308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741312+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:54:39.741312+0000","last_peered":"2023-11-03T13:54:39.741312+0000","last_clean":"2023-11-03T13:54:39.741312+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:54:39.741312+0000","last_undegraded":"2023-11-03T13:54:39.741312+0000","last_fullsized":"2023-11-03T13:54:39.741312+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":602632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.32000000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45100000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.29599999999999999,"15min":0.32200000000000001},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46800000000000003,"5min":0.5,"15min":0.5},"last":0.497}]},{"osd":1,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.31900000000000001},"min":{"1min":0.26600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41499999999999998,"5min":0.45800000000000002,"15min":0.53900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.55100000000000005},"last":0.47999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738461,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.30299999999999999},"min":{"1min":0.23400000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.307,"15min":0.32700000000000001},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.498,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36199999999999999}]},{"osd":2,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.29699999999999999,"15min":0.33700000000000002},"min":{"1min":0.26800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.318},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.33900000000000002},"min":{"1min":0.27300000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343372,"statfs":{"total":96636764160,"available":96607612928,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.29299999999999998,"15min":0.309},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46899999999999997,"5min":0.505,"15min":0.505},"last":0.246},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.308,"15min":0.316},"min":{"1min":0.27500000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.29599999999999999,"15min":0.318},"min":{"1min":0.24199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.27000000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:57:01.973 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-03T13:57:02.258 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:02.258 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":143,"fsid":"3628883d-b18e-4d59-aee7-61c0fc38535d","created":"2023-11-03T13:49:59.452451+0000","modified":"2023-11-03T13:54:38.473136+0000","last_up_change":"2023-11-03T13:50:08.553242+0000","last_in_change":"2023-11-03T13:50:01.400625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-03T13:50:07.579756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-03T13:50:08.976467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47768af5-4f35-4087-bb4b-e0a73a13a74b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6801","nonce":1963686469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6803","nonce":1963686469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6807","nonce":1963686469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":1963686469},{"type":"v1","addr":"172.21.15.73:6805","nonce":1963686469}]},"public_addr":"172.21.15.73:6801/1963686469","cluster_addr":"172.21.15.73:6803/1963686469","heartbeat_back_addr":"172.21.15.73:6807/1963686469","heartbeat_front_addr":"172.21.15.73:6805/1963686469","state":["exists","up"]},{"osd":1,"uuid":"8fbd18a9-0046-46d9-aae9-f460f3751db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6809","nonce":910736944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6811","nonce":910736944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6815","nonce":910736944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":910736944},{"type":"v1","addr":"172.21.15.73:6813","nonce":910736944}]},"public_addr":"172.21.15.73:6809/910736944","cluster_addr":"172.21.15.73:6811/910736944","heartbeat_back_addr":"172.21.15.73:6815/910736944","heartbeat_front_addr":"172.21.15.73:6813/910736944","state":["exists","up"]},{"osd":2,"uuid":"8e528929-d22b-4938-8bdc-ce337fa128ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6817","nonce":1497194653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6819","nonce":1497194653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6823","nonce":1497194653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1497194653},{"type":"v1","addr":"172.21.15.73:6821","nonce":1497194653}]},"public_addr":"172.21.15.73:6817/1497194653","cluster_addr":"172.21.15.73:6819/1497194653","heartbeat_back_addr":"172.21.15.73:6823/1497194653","heartbeat_front_addr":"172.21.15.73:6821/1497194653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.494074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.432141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-03T13:50:06.581343+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-03T13:57:03.272 INFO:tasks.ceph:Scrubbing osd.0 2023-11-03T13:57:03.273 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-03T13:57:03.610 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2023-11-03T13:57:03.611 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-03T13:57:03.611 INFO:teuthology.orchestra.run.smithi073.stdout:} 2023-11-03T13:57:03.621 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-03T13:57:03.896 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 0 to deep-scrub 2023-11-03T13:57:03.909 INFO:tasks.ceph:Scrubbing osd.1 2023-11-03T13:57:03.909 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-03T13:57:04.030 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2023-11-03T13:57:04.030 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-03T13:57:04.031 INFO:teuthology.orchestra.run.smithi073.stdout:} 2023-11-03T13:57:04.040 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-03T13:57:04.346 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 1 to deep-scrub 2023-11-03T13:57:04.359 INFO:tasks.ceph:Scrubbing osd.2 2023-11-03T13:57:04.360 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-03T13:57:04.482 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2023-11-03T13:57:04.483 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-03T13:57:04.483 INFO:teuthology.orchestra.run.smithi073.stdout:} 2023-11-03T13:57:04.493 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-03T13:57:04.770 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 2 to deep-scrub 2023-11-03T13:57:04.783 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:57:05.057 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:05.058 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:57:05.069 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2023-11-03T13:57:03.609587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89504,"kb_used_data":4268,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283026016,"statfs":{"total":289910292480,"available":289818640384,"internally_reserved":0,"allocated":4370432,"data_stored":2741808,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002022"},"pg_stats":[{"pgid":"1.0","version":"10'97","reported_seq":266,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741248+0000","last_change":"2023-11-03T13:50:09.570334+0000","last_active":"2023-11-03T13:54:39.741248+0000","last_peered":"2023-11-03T13:54:39.741248+0000","last_clean":"2023-11-03T13:54:39.741248+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:54:39.741248+0000","last_undegraded":"2023-11-03T13:54:39.741248+0000","last_fullsized":"2023-11-03T13:54:39.741248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:08.553242+0000","last_clean_scrub_stamp":"2023-11-03T13:50:08.553242+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:27:26.168481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741246+0000","last_change":"2023-11-03T13:50:10.570276+0000","last_active":"2023-11-03T13:54:39.741246+0000","last_peered":"2023-11-03T13:54:39.741246+0000","last_clean":"2023-11-03T13:54:39.741246+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:54:39.741246+0000","last_undegraded":"2023-11-03T13:54:39.741246+0000","last_fullsized":"2023-11-03T13:54:39.741246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:48:07.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478412+0000","last_change":"2023-11-03T13:50:10.569159+0000","last_active":"2023-11-03T13:54:38.478412+0000","last_peered":"2023-11-03T13:54:38.478412+0000","last_clean":"2023-11-03T13:54:38.478412+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:54:38.478412+0000","last_undegraded":"2023-11-03T13:54:38.478412+0000","last_fullsized":"2023-11-03T13:54:38.478412+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:30:58.654719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478399+0000","last_change":"2023-11-03T13:50:10.568562+0000","last_active":"2023-11-03T13:54:38.478399+0000","last_peered":"2023-11-03T13:54:38.478399+0000","last_clean":"2023-11-03T13:54:38.478399+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:54:38.478399+0000","last_undegraded":"2023-11-03T13:54:38.478399+0000","last_fullsized":"2023-11-03T13:54:38.478399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T01:19:18.640229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:38.478406+0000","last_change":"2023-11-03T13:50:10.570205+0000","last_active":"2023-11-03T13:54:38.478406+0000","last_peered":"2023-11-03T13:54:38.478406+0000","last_clean":"2023-11-03T13:54:38.478406+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:54:38.478406+0000","last_undegraded":"2023-11-03T13:54:38.478406+0000","last_fullsized":"2023-11-03T13:54:38.478406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:19:11.240778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741271+0000","last_change":"2023-11-03T13:50:10.572615+0000","last_active":"2023-11-03T13:54:39.741271+0000","last_peered":"2023-11-03T13:54:39.741271+0000","last_clean":"2023-11-03T13:54:39.741271+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:54:39.741271+0000","last_undegraded":"2023-11-03T13:54:39.741271+0000","last_fullsized":"2023-11-03T13:54:39.741271+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T16:17:56.683025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741298+0000","last_change":"2023-11-03T13:50:10.571603+0000","last_active":"2023-11-03T13:54:39.741298+0000","last_peered":"2023-11-03T13:54:39.741298+0000","last_clean":"2023-11-03T13:54:39.741298+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:54:39.741298+0000","last_undegraded":"2023-11-03T13:54:39.741298+0000","last_fullsized":"2023-11-03T13:54:39.741298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:19:09.988411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741308+0000","last_change":"2023-11-03T13:50:10.571815+0000","last_active":"2023-11-03T13:54:39.741308+0000","last_peered":"2023-11-03T13:54:39.741308+0000","last_clean":"2023-11-03T13:54:39.741308+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:54:39.741308+0000","last_undegraded":"2023-11-03T13:54:39.741308+0000","last_fullsized":"2023-11-03T13:54:39.741308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:44:19.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:54:39.741312+0000","last_change":"2023-11-03T13:50:10.572989+0000","last_active":"2023-11-03T13:54:39.741312+0000","last_peered":"2023-11-03T13:54:39.741312+0000","last_clean":"2023-11-03T13:54:39.741312+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:54:39.741312+0000","last_undegraded":"2023-11-03T13:54:39.741312+0000","last_fullsized":"2023-11-03T13:54:39.741312+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:50:09.557992+0000","last_clean_scrub_stamp":"2023-11-03T13:50:09.557992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T18:00:39.551450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":602632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.32000000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45100000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.29599999999999999,"15min":0.32200000000000001},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.46800000000000003,"5min":0.5,"15min":0.5},"last":0.497}]},{"osd":1,"last update":"Fri Nov 3 13:56:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.31900000000000001},"min":{"1min":0.26600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41499999999999998,"5min":0.45800000000000002,"15min":0.53900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.55100000000000005},"last":0.47999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.30299999999999999},"min":{"1min":0.23400000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.247},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.307,"15min":0.32700000000000001},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.498,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Fri Nov 3 13:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.29699999999999999,"15min":0.33700000000000002},"min":{"1min":0.26800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.33900000000000002},"min":{"1min":0.27300000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343372,"statfs":{"total":96636764160,"available":96607612928,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.29299999999999998,"15min":0.309},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46899999999999997,"5min":0.505,"15min":0.505},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.308,"15min":0.316},"min":{"1min":0.27500000000000002,"5min":0.186,"15min":0.186},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.22600000000000001}]},{"osd":2,"last update":"Fri Nov 3 13:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.29599999999999999,"15min":0.318},"min":{"1min":0.24199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.27000000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:57:05.072 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-03T13:50:08.553242+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=8, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.073 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.074 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.074 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.074 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.075 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.075 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.076 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.076 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-03T13:50:09.557992+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=50, tm_sec=9, tm_wday=4, tm_yday=307, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=3, tm_hour=13, tm_min=57, tm_sec=2, tm_wday=4, tm_yday=307, tm_isdst=0) 2023-11-03T13:57:05.076 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-03T13:57:25.078 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-03T13:57:25.362 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-11-03T13:57:25.363 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2023-11-03T13:57:25.374 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2023-11-03T13:57:23.613221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":89476,"kb_used_data":4268,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283026044,"statfs":{"total":289910292480,"available":289818669056,"internally_reserved":0,"allocated":4370432,"data_stored":2741808,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":4782,"internal_metadata":87223634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002440"},"pg_stats":[{"pgid":"1.0","version":"143'99","reported_seq":277,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:09.349874+0000","last_change":"2023-11-03T13:57:09.349940+0000","last_active":"2023-11-03T13:57:09.349874+0000","last_peered":"2023-11-03T13:57:09.349874+0000","last_clean":"2023-11-03T13:57:09.349874+0000","last_became_active":"2023-11-03T13:50:09.570051+0000","last_became_peered":"2023-11-03T13:50:09.570051+0000","last_unstale":"2023-11-03T13:57:09.349874+0000","last_undegraded":"2023-11-03T13:57:09.349874+0000","last_fullsized":"2023-11-03T13:57:09.349874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'99","last_scrub_stamp":"2023-11-03T13:57:09.349790+0000","last_deep_scrub":"143'99","last_deep_scrub_stamp":"2023-11-03T13:57:09.349790+0000","last_clean_scrub_stamp":"2023-11-03T13:57:09.349790+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T20:05:56.814451+0000","scrub_duration":0.0048353229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:13.247927+0000","last_change":"2023-11-03T13:57:13.247969+0000","last_active":"2023-11-03T13:57:13.247927+0000","last_peered":"2023-11-03T13:57:13.247927+0000","last_clean":"2023-11-03T13:57:13.247927+0000","last_became_active":"2023-11-03T13:50:10.569205+0000","last_became_peered":"2023-11-03T13:50:10.569205+0000","last_unstale":"2023-11-03T13:57:13.247927+0000","last_undegraded":"2023-11-03T13:57:13.247927+0000","last_fullsized":"2023-11-03T13:57:13.247927+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:13.247828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:13.247828+0000","last_clean_scrub_stamp":"2023-11-03T13:57:13.247828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T23:36:37.430192+0000","scrub_duration":0.0014049710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:07.406985+0000","last_change":"2023-11-03T13:57:07.407031+0000","last_active":"2023-11-03T13:57:07.406985+0000","last_peered":"2023-11-03T13:57:07.406985+0000","last_clean":"2023-11-03T13:57:07.406985+0000","last_became_active":"2023-11-03T13:50:10.568954+0000","last_became_peered":"2023-11-03T13:50:10.568954+0000","last_unstale":"2023-11-03T13:57:07.406985+0000","last_undegraded":"2023-11-03T13:57:07.406985+0000","last_fullsized":"2023-11-03T13:57:07.406985+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:07.406891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:07.406891+0000","last_clean_scrub_stamp":"2023-11-03T13:57:07.406891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T14:48:41.514250+0000","scrub_duration":0.0015043140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:06.357012+0000","last_change":"2023-11-03T13:57:06.357056+0000","last_active":"2023-11-03T13:57:06.357012+0000","last_peered":"2023-11-03T13:57:06.357012+0000","last_clean":"2023-11-03T13:57:06.357012+0000","last_became_active":"2023-11-03T13:50:10.568293+0000","last_became_peered":"2023-11-03T13:50:10.568293+0000","last_unstale":"2023-11-03T13:57:06.357012+0000","last_undegraded":"2023-11-03T13:57:06.357012+0000","last_fullsized":"2023-11-03T13:57:06.357012+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:06.356936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:06.356936+0000","last_clean_scrub_stamp":"2023-11-03T13:57:06.356936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:54:47.446238+0000","scrub_duration":0.0013425799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":146,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:05.822062+0000","last_change":"2023-11-03T13:57:05.822104+0000","last_active":"2023-11-03T13:57:05.822062+0000","last_peered":"2023-11-03T13:57:05.822062+0000","last_clean":"2023-11-03T13:57:05.822062+0000","last_became_active":"2023-11-03T13:50:10.569943+0000","last_became_peered":"2023-11-03T13:50:10.569943+0000","last_unstale":"2023-11-03T13:57:05.822062+0000","last_undegraded":"2023-11-03T13:57:05.822062+0000","last_fullsized":"2023-11-03T13:57:05.822062+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:05.821985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:05.821985+0000","last_clean_scrub_stamp":"2023-11-03T13:57:05.821985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T23:36:51.672750+0000","scrub_duration":0.0013351509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:21.198195+0000","last_change":"2023-11-03T13:57:21.198276+0000","last_active":"2023-11-03T13:57:21.198195+0000","last_peered":"2023-11-03T13:57:21.198195+0000","last_clean":"2023-11-03T13:57:21.198195+0000","last_became_active":"2023-11-03T13:50:10.572198+0000","last_became_peered":"2023-11-03T13:50:10.572198+0000","last_unstale":"2023-11-03T13:57:21.198195+0000","last_undegraded":"2023-11-03T13:57:21.198195+0000","last_fullsized":"2023-11-03T13:57:21.198195+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:21.198092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:21.198092+0000","last_clean_scrub_stamp":"2023-11-03T13:57:21.198092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T00:37:13.410669+0000","scrub_duration":0.001421744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:17.248305+0000","last_change":"2023-11-03T13:57:17.248349+0000","last_active":"2023-11-03T13:57:17.248305+0000","last_peered":"2023-11-03T13:57:17.248305+0000","last_clean":"2023-11-03T13:57:17.248305+0000","last_became_active":"2023-11-03T13:50:10.571329+0000","last_became_peered":"2023-11-03T13:50:10.571329+0000","last_unstale":"2023-11-03T13:57:17.248305+0000","last_undegraded":"2023-11-03T13:57:17.248305+0000","last_fullsized":"2023-11-03T13:57:17.248305+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:17.248208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:17.248208+0000","last_clean_scrub_stamp":"2023-11-03T13:57:17.248208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-05T00:25:49.271113+0000","scrub_duration":0.001570306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:10.302705+0000","last_change":"2023-11-03T13:57:10.302752+0000","last_active":"2023-11-03T13:57:10.302705+0000","last_peered":"2023-11-03T13:57:10.302705+0000","last_clean":"2023-11-03T13:57:10.302705+0000","last_became_active":"2023-11-03T13:50:10.571656+0000","last_became_peered":"2023-11-03T13:50:10.571656+0000","last_unstale":"2023-11-03T13:57:10.302705+0000","last_undegraded":"2023-11-03T13:57:10.302705+0000","last_fullsized":"2023-11-03T13:57:10.302705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:10.302622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:10.302622+0000","last_clean_scrub_stamp":"2023-11-03T13:57:10.302622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T22:57:38.636079+0000","scrub_duration":0.001346968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-11-03T13:57:14.275451+0000","last_change":"2023-11-03T13:57:14.275518+0000","last_active":"2023-11-03T13:57:14.275451+0000","last_peered":"2023-11-03T13:57:14.275451+0000","last_clean":"2023-11-03T13:57:14.275451+0000","last_became_active":"2023-11-03T13:50:10.572706+0000","last_became_peered":"2023-11-03T13:50:10.572706+0000","last_unstale":"2023-11-03T13:57:14.275451+0000","last_undegraded":"2023-11-03T13:57:14.275451+0000","last_fullsized":"2023-11-03T13:57:14.275451+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-03T13:57:14.275349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-03T13:57:14.275349+0000","last_clean_scrub_stamp":"2023-11-03T13:57:14.275349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-04T14:37:17.215303+0000","scrub_duration":0.00166216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":933912,"data_compressed":48340,"data_compressed_allocated":65536,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1421312,"data_stored":602632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.32000000000000001},"min":{"1min":0.223,"5min":0.183,"15min":0.17899999999999999},"max":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.318,"15min":0.32500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":0.497,"5min":0.5,"15min":0.5},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Nov 3 13:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.31900000000000001},"min":{"1min":0.23999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":0.45800000000000002,"15min":0.53900000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.32300000000000001,"15min":0.35099999999999998},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.47999999999999998,"5min":0.52100000000000002,"15min":0.55100000000000005},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359738466,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343384,"statfs":{"total":96636764160,"available":96607625216,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 3 13:57:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.28799999999999998,"15min":0.30099999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.41399999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.23400000000000001,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":0.45200000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.26600000000000001}]},{"osd":2,"last update":"Fri Nov 3 13:57:23 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.29799999999999999,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.19900000000000001},"max":{"1min":0.46600000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.248},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.307,"15min":0.33700000000000002},"min":{"1min":0.215,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.43099999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343384,"statfs":{"total":96636764160,"available":96607625216,"internally_reserved":0,"allocated":1474560,"data_stored":1069588,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 3 13:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.30499999999999999,"15min":0.311},"min":{"1min":0.21299999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.505},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"min":{"1min":0.22600000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.47199999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Fri Nov 3 13:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.309,"15min":0.31900000000000001},"min":{"1min":0.26000000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33700000000000002},"min":{"1min":0.28699999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.314}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":466956,"data_compressed":24170,"data_compressed_allocated":32768,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-03T13:57:25.376 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-03T13:57:25.672 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-03T13:57:25.672 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-03T13:57:25.673 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-03T13:57:25.673 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.721 INFO:tasks.ceph.osd.0:Stopped 2023-11-03T13:57:25.722 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-03T13:57:25.722 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.759 INFO:tasks.ceph.osd.1:Stopped 2023-11-03T13:57:25.760 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-03T13:57:25.760 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.831 INFO:tasks.ceph.osd.2:Stopped 2023-11-03T13:57:25.831 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-03T13:57:25.832 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-03T13:57:25.832 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.894 INFO:tasks.ceph.mgr.x:Stopped 2023-11-03T13:57:25.895 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-03T13:57:25.895 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-03T13:57:25.896 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.946 INFO:tasks.ceph.mon.a:Stopped 2023-11-03T13:57:25.946 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-03T13:57:25.947 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:25.996 INFO:tasks.ceph.mon.b:Stopped 2023-11-03T13:57:25.997 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-03T13:57:25.997 INFO:teuthology.orchestra.run:waiting for 300 2023-11-03T13:57:26.047 INFO:tasks.ceph.mon.c:Stopped 2023-11-03T13:57:26.048 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-03T13:57:26.048 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-11-03T13:57:26.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:57:26.129 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-03T13:57:26.308 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:57:26.309 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-03T13:57:26.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:57:26.407 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-03T13:57:26.499 INFO:tasks.ceph:Archiving mon data... 2023-11-03T13:57:26.500 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/data/mon.a.tgz 2023-11-03T13:57:26.501 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-11-03T13:57:26.517 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.7HNcfPbwkm 2023-11-03T13:57:26.517 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7HNcfPbwkm 2023-11-03T13:57:26.659 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.7HNcfPbwkm 2023-11-03T13:57:26.747 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.7HNcfPbwkm is 288KB 2023-11-03T13:57:26.841 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.7HNcfPbwkm 2023-11-03T13:57:26.857 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/data/mon.b.tgz 2023-11-03T13:57:26.857 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-11-03T13:57:26.908 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.fzjC3Im0Fb 2023-11-03T13:57:26.909 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.fzjC3Im0Fb 2023-11-03T13:57:27.031 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.fzjC3Im0Fb 2023-11-03T13:57:27.115 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.fzjC3Im0Fb is 288KB 2023-11-03T13:57:27.209 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.fzjC3Im0Fb 2023-11-03T13:57:27.224 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/data/mon.c.tgz 2023-11-03T13:57:27.225 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-11-03T13:57:27.276 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.MH5dL9t5hG 2023-11-03T13:57:27.277 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MH5dL9t5hG 2023-11-03T13:57:27.398 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.MH5dL9t5hG 2023-11-03T13:57:27.481 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.MH5dL9t5hG is 288KB 2023-11-03T13:57:27.576 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.MH5dL9t5hG 2023-11-03T13:57:27.591 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-03T13:57:27.591 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-03T13:57:27.657 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-03T13:57:27.658 DEBUG:teuthology.orchestra.run.smithi073:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-03T13:57:27.726 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-03T13:57:27.730 INFO:tasks.ceph:Archiving crash dumps... 2023-11-03T13:57:27.732 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/remote/smithi073/crash 2023-11-03T13:57:27.733 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-03T13:57:27.796 INFO:tasks.ceph:Compressing logs... 2023-11-03T13:57:27.797 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-03T13:57:27.987 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90566.log 2023-11-03T13:57:27.987 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96005.log 2023-11-03T13:57:27.987 INFO:teuthology.orchestra.run.smithi073.stderr:gzip/var/log/ceph/ceph-client.admin.90566.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.92719.log 2023-11-03T13:57:27.988 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90566.log.gz 2023-11-03T13:57:27.988 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91460.log 2023-11-03T13:57:27.988 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96005.log.gz 2023-11-03T13:57:27.989 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93611.log 2023-11-03T13:57:27.989 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92719.log.gz 2023-11-03T13:57:27.989 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91020.log 2023-11-03T13:57:27.989 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91460.log.gz 2023-11-03T13:57:27.990 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89347.log 2023-11-03T13:57:27.990 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93611.log.gz 2023-11-03T13:57:27.990 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93386.log 2023-11-03T13:57:27.990 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91020.log.gz 2023-11-03T13:57:27.991 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89347.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-03T13:57:27.991 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89347.log.gz 2023-11-03T13:57:27.991 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88599.log 2023-11-03T13:57:27.991 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93386.log.gz 2023-11-03T13:57:27.992 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.xx-profile-rd.95493.log 2023-11-03T13:57:27.992 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93586.log 2023-11-03T13:57:27.992 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88599.log.gz 2023-11-03T13:57:27.993 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87509.log 2023-11-03T13:57:27.993 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95493.log.gz 2023-11-03T13:57:27.993 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90972.log 2023-11-03T13:57:27.993 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93586.log.gz 2023-11-03T13:57:27.994 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91387.log 2023-11-03T13:57:27.994 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87509.log.gz 2023-11-03T13:57:27.994 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90038.log 2023-11-03T13:57:27.994 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90972.log.gz 2023-11-03T13:57:27.995 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88992.log 2023-11-03T13:57:27.995 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91387.log.gz 2023-11-03T13:57:27.995 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87801.log 2023-11-03T13:57:27.995 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90038.log.gz 2023-11-03T13:57:27.996 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91900.log 2023-11-03T13:57:27.996 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88992.log.gz 2023-11-03T13:57:27.996 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89735.log 2023-11-03T13:57:27.996 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87801.log.gz 2023-11-03T13:57:27.997 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd2.95728.log 2023-11-03T13:57:27.997 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91900.log.gz 2023-11-03T13:57:27.997 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88500.log 2023-11-03T13:57:27.998 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89735.log.gz 2023-11-03T13:57:27.998 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd2.95728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.95113.log 2023-11-03T13:57:27.998 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd2.95728.log.gz 2023-11-03T13:57:27.998 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91731.log 2023-11-03T13:57:27.999 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88500.log.gz 2023-11-03T13:57:27.999 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.95042.log 2023-11-03T13:57:27.999 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95113.log.gz 2023-11-03T13:57:27.999 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93356.log 2023-11-03T13:57:28.000 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91731.log.gz 2023-11-03T13:57:28.000 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92117.log 2023-11-03T13:57:28.000 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95042.log.gz 2023-11-03T13:57:28.000 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96246.log 2023-11-03T13:57:28.001 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93356.log.gz 2023-11-03T13:57:28.001 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90062.log 2023-11-03T13:57:28.001 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92117.log.gz 2023-11-03T13:57:28.001 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91777.log 2023-11-03T13:57:28.002 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96246.log.gz 2023-11-03T13:57:28.002 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90062.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.90062.log.gz 2023-11-03T13:57:28.002 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.89543.log 2023-11-03T13:57:28.003 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89569.log 2023-11-03T13:57:28.003 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91777.log.gz 2023-11-03T13:57:28.003 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89925.log 2023-11-03T13:57:28.003 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89543.log.gz 2023-11-03T13:57:28.004 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89397.log 2023-11-03T13:57:28.004 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89569.log.gz 2023-11-03T13:57:28.004 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90596.log 2023-11-03T13:57:28.004 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89925.log.gz 2023-11-03T13:57:28.005 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94533.log 2023-11-03T13:57:28.005 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89397.log.gz 2023-11-03T13:57:28.005 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89421.log 2023-11-03T13:57:28.005 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90596.log.gz 2023-11-03T13:57:28.006 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91191.log 2023-11-03T13:57:28.006 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94533.log.gz 2023-11-03T13:57:28.006 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91239.log 2023-11-03T13:57:28.006 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89421.log.gz 2023-11-03T13:57:28.007 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91339.log 2023-11-03T13:57:28.007 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91191.log.gz 2023-11-03T13:57:28.007 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89447.log 2023-11-03T13:57:28.007 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91239.log.gz 2023-11-03T13:57:28.008 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95159.log 2023-11-03T13:57:28.008 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91339.log.gz 2023-11-03T13:57:28.008 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92341.log 2023-11-03T13:57:28.009 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89447.log.gz 2023-11-03T13:57:28.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92913.log 2023-11-03T13:57:28.009 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95159.log.gz 2023-11-03T13:57:28.009 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96270.log 2023-11-03T13:57:28.010 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92341.log.gz 2023-11-03T13:57:28.010 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94193.log 2023-11-03T13:57:28.010 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92913.log.gz 2023-11-03T13:57:28.010 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91948.log 2023-11-03T13:57:28.011 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96270.log.gz 2023-11-03T13:57:28.011 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94169.log 2023-11-03T13:57:28.011 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94193.log.gz 2023-11-03T13:57:28.011 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94780.log 0.0% 2023-11-03T13:57:28.012 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91948.log.gz 2023-11-03T13:57:28.012 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92018.log 2023-11-03T13:57:28.012 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94169.log.gz 2023-11-03T13:57:28.013 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89038.log 2023-11-03T13:57:28.013 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94780.log.gz 2023-11-03T13:57:28.013 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92018.log.gz 2023-11-03T13:57:28.013 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90996.log 2023-11-03T13:57:28.014 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92556.log 2023-11-03T13:57:28.014 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89038.log.gz 2023-11-03T13:57:28.014 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93162.log 2023-11-03T13:57:28.014 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90996.log.gz 2023-11-03T13:57:28.015 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93791.log 2023-11-03T13:57:28.015 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92556.log.gz 2023-11-03T13:57:28.015 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94898.log 2023-11-03T13:57:28.015 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93162.log.gz 2023-11-03T13:57:28.016 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95540.log 2023-11-03T13:57:28.016 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93791.log.gz 2023-11-03T13:57:28.016 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96934.log 2023-11-03T13:57:28.017 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94898.log.gz 2023-11-03T13:57:28.017 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84696.log 2023-11-03T13:57:28.017 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95540.log.gz 2023-11-03T13:57:28.017 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91972.log 2023-11-03T13:57:28.018 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96934.log.gz 2023-11-03T13:57:28.018 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84696.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.84696.log.gz --verbose 2023-11-03T13:57:28.018 INFO:teuthology.orchestra.run.smithi073.stderr: -- /var/log/ceph/ceph-client.admin.88525.log 2023-11-03T13:57:28.018 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95207.log 2023-11-03T13:57:28.019 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91972.log.gz 2023-11-03T13:57:28.019 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95802.log 2023-11-03T13:57:28.019 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88525.log.gz 2023-11-03T13:57:28.019 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92937.log 2023-11-03T13:57:28.020 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95207.log.gz 2023-11-03T13:57:28.020 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91825.log 2023-11-03T13:57:28.020 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95802.log.gz 2023-11-03T13:57:28.021 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92577.log 2023-11-03T13:57:28.021 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92937.log.gz 2023-11-03T13:57:28.021 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95327.log 2023-11-03T13:57:28.021 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91825.log.gz 2023-11-03T13:57:28.022 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94558.log 2023-11-03T13:57:28.022 INFO:teuthology.orchestra.run.smithi073.stderr: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.92577.log.gz 2023-11-03T13:57:28.022 INFO:teuthology.orchestra.run.smithi073.stderr:gzip/var/log/ceph/ceph-client.xx-profile-rw.95327.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.94435.log 2023-11-03T13:57:28.022 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95327.log.gz 2023-11-03T13:57:28.023 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93538.log 2023-11-03T13:57:28.023 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94558.log.gz 2023-11-03T13:57:28.023 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90926.log 2023-11-03T13:57:28.023 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94435.log.gz 2023-11-03T13:57:28.024 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92646.log 2023-11-03T13:57:28.024 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93538.log.gz 2023-11-03T13:57:28.024 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93865.log 2023-11-03T13:57:28.024 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90926.log.gz 2023-11-03T13:57:28.025 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90384.log 2023-11-03T13:57:28.025 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92646.log.gz 2023-11-03T13:57:28.025 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92093.log 2023-11-03T13:57:28.026 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93865.log.gz 2023-11-03T13:57:28.026 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89015.log 2023-11-03T13:57:28.026 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90384.log.gz 2023-11-03T13:57:28.026 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95752.log 2023-11-03T13:57:28.027 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92093.log.gz 2023-11-03T13:57:28.027 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96762.log 2023-11-03T13:57:28.027 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89015.log.gz 2023-11-03T13:57:28.027 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93687.log 2023-11-03T13:57:28.028 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95752.log.gz 2023-11-03T13:57:28.028 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92514.log 0.0% 2023-11-03T13:57:28.028 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96762.log.gz 2023-11-03T13:57:28.028 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96174.log 2023-11-03T13:57:28.029 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93687.log.gz 2023-11-03T13:57:28.029 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94757.log 2023-11-03T13:57:28.029 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92514.log.gz 2023-11-03T13:57:28.030 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91486.log 2023-11-03T13:57:28.030 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96174.log.gz 2023-11-03T13:57:28.030 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88967.log 2023-11-03T13:57:28.030 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94757.log.gz 2023-11-03T13:57:28.031 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95563.log 2023-11-03T13:57:28.031 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91486.log.gz 2023-11-03T13:57:28.031 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96507.log 2023-11-03T13:57:28.031 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88967.log: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.88967.log.gz 2023-11-03T13:57:28.032 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92316.log 2023-11-03T13:57:28.032 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95563.log.gz 2023-11-03T13:57:28.032 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90141.log 2023-11-03T13:57:28.032 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96507.log.gz 2023-11-03T13:57:28.033 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91264.log 2023-11-03T13:57:28.033 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92316.log.gz 2023-11-03T13:57:28.033 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96078.log 2023-11-03T13:57:28.033 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90141.log.gz 2023-11-03T13:57:28.034 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93890.log 2023-11-03T13:57:28.034 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91264.log.gz 2023-11-03T13:57:28.034 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96078.log.gz 2023-11-03T13:57:28.035 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93712.log 2023-11-03T13:57:28.035 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93890.log.gz 2023-11-03T13:57:28.035 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93010.log 2023-11-03T13:57:28.035 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93110.log 2023-11-03T13:57:28.036 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93712.log.gz 2023-11-03T13:57:28.036 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92622.log 2023-11-03T13:57:28.036 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93010.log.gz 2023-11-03T13:57:28.036 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90878.log 2023-11-03T13:57:28.037 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93110.log.gz 2023-11-03T13:57:28.037 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96804.log 2023-11-03T13:57:28.037 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92622.log.gz 2023-11-03T13:57:28.037 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90878.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.90878.log.gz 2023-11-03T13:57:28.038 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95680.log 2023-11-03T13:57:28.038 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92292.log 2023-11-03T13:57:28.038 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96804.log.gz 2023-11-03T13:57:28.039 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88181.log 2023-11-03T13:57:28.039 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95680.log.gz 2023-11-03T13:57:28.039 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93416.log 2023-11-03T13:57:28.039 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92292.log.gz 2023-11-03T13:57:28.040 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94217.log 2023-11-03T13:57:28.040 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88181.log.gz 2023-11-03T13:57:28.040 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95446.log 2023-11-03T13:57:28.040 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93416.log.gz 2023-11-03T13:57:28.041 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90902.log 2023-11-03T13:57:28.041 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94217.log.gz 2023-11-03T13:57:28.041 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93817.log 2023-11-03T13:57:28.041 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95446.log.gz 2023-11-03T13:57:28.042 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90902.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-03T13:57:28.042 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90902.log.gz 2023-11-03T13:57:28.042 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89521.log 2023-11-03T13:57:28.042 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93817.log.gz 2023-11-03T13:57:28.043 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87310.log 2023-11-03T13:57:28.043 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.89521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96294.log 2023-11-03T13:57:28.043 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89521.log.gz 2023-11-03T13:57:28.044 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94922.log 2023-11-03T13:57:28.044 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87310.log.gz 2023-11-03T13:57:28.044 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92440.log 2023-11-03T13:57:28.044 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96294.log.gz 2023-11-03T13:57:28.045 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87800.log 2023-11-03T13:57:28.045 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94922.log.gz 2023-11-03T13:57:28.045 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95303.log 2023-11-03T13:57:28.046 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92440.log.gz 2023-11-03T13:57:28.046 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89173.log 2023-11-03T13:57:28.046 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87800.log.gz 2023-11-03T13:57:28.046 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88918.log 2023-11-03T13:57:28.047 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95303.log.gz 2023-11-03T13:57:28.047 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94584.log 2023-11-03T13:57:28.047 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89173.log.gz 2023-11-03T13:57:28.047 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90014.log 2023-11-03T13:57:28.048 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88918.log.gz 2023-11-03T13:57:28.048 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91363.log 2023-11-03T13:57:28.048 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94584.log.gz 2023-11-03T13:57:28.048 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95852.log 2023-11-03T13:57:28.049 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90014.log.gz 2023-11-03T13:57:28.049 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93490.log 2023-11-03T13:57:28.049 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91363.log.gz 2023-11-03T13:57:28.049 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92365.log 2023-11-03T13:57:28.050 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95852.log.gz 2023-11-03T13:57:28.050 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91143.log 2023-11-03T13:57:28.050 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93490.log.gz 2023-11-03T13:57:28.051 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90762.log 2023-11-03T13:57:28.051 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92365.log.gz 2023-11-03T13:57:28.051 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93739.log 2023-11-03T13:57:28.051 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91143.log.gz 2023-11-03T13:57:28.052 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94707.log 2023-11-03T13:57:28.052 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90762.log.gz 2023-11-03T13:57:28.052 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88450.log 2023-11-03T13:57:28.052 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93739.log.gz 2023-11-03T13:57:28.053 INFO:teuthology.orchestra.run.smithi073.stderr: 96.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-03T13:57:28.053 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92416.log 2023-11-03T13:57:28.053 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94707.log.gz 2023-11-03T13:57:28.053 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95351.log 2023-11-03T13:57:28.054 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88450.log.gz 2023-11-03T13:57:28.054 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88673.log 2023-11-03T13:57:28.054 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92416.log.gz 2023-11-03T13:57:28.055 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93967.log 2023-11-03T13:57:28.055 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95351.log.gz 2023-11-03T13:57:28.055 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92598.log 2023-11-03T13:57:28.055 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88673.log.gz 2023-11-03T13:57:28.056 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87887.log 2023-11-03T13:57:28.056 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93967.log.gz 2023-11-03T13:57:28.056 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89321.log 2023-11-03T13:57:28.056 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92598.log.gz 2023-11-03T13:57:28.057 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87551.log 2023-11-03T13:57:28.057 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87887.log.gz 2023-11-03T13:57:28.057 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90432.log 2023-11-03T13:57:28.057 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89321.log.gz 2023-11-03T13:57:28.058 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87409.log 2023-11-03T13:57:28.058 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87551.log.gz 2023-11-03T13:57:28.058 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89473.log 2023-11-03T13:57:28.058 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90432.log.gz 2023-11-03T13:57:28.059 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90250.log 2023-11-03T13:57:28.059 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87409.log.gz 2023-11-03T13:57:28.059 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92889.log 2023-11-03T13:57:28.060 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89473.log.gz 2023-11-03T13:57:28.060 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94334.log 2023-11-03T13:57:28.060 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90250.log.gz 2023-11-03T13:57:28.060 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94509.log 2023-11-03T13:57:28.061 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92889.log.gz 2023-11-03T13:57:28.061 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94359.log 2023-11-03T13:57:28.061 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94334.log.gz 2023-11-03T13:57:28.061 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90086.log 2023-11-03T13:57:28.062 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94509.log.gz 2023-11-03T13:57:28.062 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96102.log 2023-11-03T13:57:28.062 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94359.log.gz 2023-11-03T13:57:28.062 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84758.log 2023-11-03T13:57:28.063 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90086.log.gz 2023-11-03T13:57:28.063 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87674.log 2023-11-03T13:57:28.063 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96102.log.gz 2023-11-03T13:57:28.064 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89647.log 2023-11-03T13:57:28.064 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84758.log.gz 2023-11-03T13:57:28.064 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96846.log 2023-11-03T13:57:28.064 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87674.log.gz 2023-11-03T13:57:28.065 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93229.log 2023-11-03T13:57:28.065 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89647.log.gz 2023-11-03T13:57:28.065 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88722.log 2023-11-03T13:57:28.065 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96846.log.gz 2023-11-03T13:57:28.066 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92745.log 2023-11-03T13:57:28.066 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93229.log.gz 2023-11-03T13:57:28.066 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91169.log 2023-11-03T13:57:28.066 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88722.log.gz 2023-11-03T13:57:28.067 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94121.log 2023-11-03T13:57:28.067 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92745.log.gz 2023-11-03T13:57:28.067 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88894.log 2023-11-03T13:57:28.068 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91169.log.gz 2023-11-03T13:57:28.068 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88223.log 2023-11-03T13:57:28.068 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94121.log.gz 2023-11-03T13:57:28.068 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.95018.log 2023-11-03T13:57:28.069 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88894.log.gz 2023-11-03T13:57:28.069 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88223.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-03T13:57:28.069 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88223.log.gz 2023-11-03T13:57:28.069 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87366.log 2023-11-03T13:57:28.070 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95018.log.gz 2023-11-03T13:57:28.070 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94803.log 2023-11-03T13:57:28.070 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95279.log 2023-11-03T13:57:28.070 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87366.log.gz 2023-11-03T13:57:28.071 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91655.log 2023-11-03T13:57:28.071 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94803.log.gz 2023-11-03T13:57:28.071 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94873.log 2023-11-03T13:57:28.071 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95279.log.gz 2023-11-03T13:57:28.072 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91536.log 2023-11-03T13:57:28.072 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91655.log.gz 2023-11-03T13:57:28.072 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96894.log 2023-11-03T13:57:28.073 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94873.log.gz 2023-11-03T13:57:28.073 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94828.log 2023-11-03T13:57:28.073 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91536.log.gz 2023-11-03T13:57:28.073 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95954.log 2023-11-03T13:57:28.074 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96894.log.gz 2023-11-03T13:57:28.074 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92842.log 2023-11-03T13:57:28.074 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94828.log.gz 2023-11-03T13:57:28.074 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89296.log 2023-11-03T13:57:28.075 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95954.log.gz 2023-11-03T13:57:28.075 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84517.log 2023-11-03T13:57:28.075 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92842.log.gz 2023-11-03T13:57:28.075 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89296.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79786.log 2023-11-03T13:57:28.076 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89296.log.gz 2023-11-03T13:57:28.076 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90226.log 2023-11-03T13:57:28.076 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84517.log.gz 2023-11-03T13:57:28.077 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.tmp-client.admin.79786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94733.log 2023-11-03T13:57:28.077 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79786.log.gz 2023-11-03T13:57:28.077 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88820.log 2023-11-03T13:57:28.077 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90226.log.gz 2023-11-03T13:57:28.078 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89805.log 2023-11-03T13:57:28.078 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94733.log.gz 2023-11-03T13:57:28.078 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92167.log 2023-11-03T13:57:28.078 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88820.log.gz 2023-11-03T13:57:28.079 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93637.log 2023-11-03T13:57:28.079 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89805.log.gz 2023-11-03T13:57:28.079 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88943.log 2023-11-03T13:57:28.079 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92167.log.gz 2023-11-03T13:57:28.080 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92196.log 2023-11-03T13:57:28.080 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93637.log.gz 2023-11-03T13:57:28.080 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90408.log 2023-11-03T13:57:28.081 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88943.log.gz 2023-11-03T13:57:28.081 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90296.log 2023-11-03T13:57:28.081 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92196.log.gz 2023-11-03T13:57:28.081 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90318.log 2023-11-03T13:57:28.082 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90408.log.gz 2023-11-03T13:57:28.082 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92694.log 2023-11-03T13:57:28.082 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90296.log.gz 2023-11-03T13:57:28.082 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90950.log 2023-11-03T13:57:28.083 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90318.log.gz 2023-11-03T13:57:28.083 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95398.log 2023-11-03T13:57:28.083 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92694.log.gz 2023-11-03T13:57:28.083 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95656.log 2023-11-03T13:57:28.084 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90950.log.gz 2023-11-03T13:57:28.084 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94286.log 2023-11-03T13:57:28.084 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95398.log.gz 2023-11-03T13:57:28.084 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93301.log 2023-11-03T13:57:28.085 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95656.log.gz 2023-11-03T13:57:28.085 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92390.log 2023-11-03T13:57:28.085 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94286.log.gz 2023-11-03T13:57:28.086 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89757.log 2023-11-03T13:57:28.086 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93301.log.gz 2023-11-03T13:57:28.086 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91582.log 2023-11-03T13:57:28.086 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92390.log.gz 2023-11-03T13:57:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93562.log 2023-11-03T13:57:28.087 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89757.log.gz 2023-11-03T13:57:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90166.log 2023-11-03T13:57:28.087 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91582.log.gz 2023-11-03T13:57:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89949.log 2023-11-03T13:57:28.088 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93562.log.gz 2023-11-03T13:57:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90166.log.gz 2023-11-03T13:57:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90854.log 2023-11-03T13:57:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89689.log 2023-11-03T13:57:28.089 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89949.log.gz 2023-11-03T13:57:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84597.log 2023-11-03T13:57:28.089 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90854.log.gz 2023-11-03T13:57:28.090 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89595.log 2023-11-03T13:57:28.090 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89689.log.gz 2023-11-03T13:57:28.090 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92267.log 2023-11-03T13:57:28.091 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84597.log.gz 2023-11-03T13:57:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89371.log 2023-11-03T13:57:28.091 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89595.log.gz 2023-11-03T13:57:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92670.log 2023-11-03T13:57:28.092 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92267.log.gz 2023-11-03T13:57:28.092 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96126.log 2023-11-03T13:57:28.092 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89371.log.gz 2023-11-03T13:57:28.092 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91705.log 2023-11-03T13:57:28.093 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92670.log.gz 2023-11-03T13:57:28.093 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95586.log 2023-11-03T13:57:28.093 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96126.log.gz 2023-11-03T13:57:28.093 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90644.log 2023-11-03T13:57:28.094 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91705.log.gz 2023-11-03T13:57:28.094 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87761.log 2023-11-03T13:57:28.094 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95586.log.gz 2023-11-03T13:57:28.094 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91681.log 2023-11-03T13:57:28.095 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90644.log.gz 2023-11-03T13:57:28.095 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88574.log 2023-11-03T13:57:28.095 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87761.log.gz 2023-11-03T13:57:28.096 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95827.log 2023-11-03T13:57:28.096 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91681.log.gz 2023-11-03T13:57:28.096 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88574.log.gz 2023-11-03T13:57:28.096 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96636.log 2023-11-03T13:57:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96551.log 2023-11-03T13:57:28.097 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95827.log.gz 2023-11-03T13:57:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92986.log 2023-11-03T13:57:28.097 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96636.log.gz 2023-11-03T13:57:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92819.log 2023-11-03T13:57:28.098 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96551.log.gz 2023-11-03T13:57:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92795.log 2023-11-03T13:57:28.098 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92986.log.gz 2023-11-03T13:57:28.099 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94409.log 2023-11-03T13:57:28.099 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92819.log.gz 2023-11-03T13:57:28.099 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91849.log 2023-11-03T13:57:28.099 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92795.log.gz 2023-11-03T13:57:28.100 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95516.log 2023-11-03T13:57:28.100 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94409.log.gz 2023-11-03T13:57:28.100 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91314.log 2023-11-03T13:57:28.101 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91849.log.gz 2023-11-03T13:57:28.101 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94659.log 2023-11-03T13:57:28.101 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95516.log.gz 2023-11-03T13:57:28.101 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92067.log 2023-11-03T13:57:28.102 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91314.log.gz 2023-11-03T13:57:28.102 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94018.log 2023-11-03T13:57:28.102 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94659.log.gz 2023-11-03T13:57:28.102 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88845.log 2023-11-03T13:57:28.103 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92067.log.gz 2023-11-03T13:57:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95231.log 2023-11-03T13:57:28.103 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94018.log.gz 2023-11-03T13:57:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90486.log 2023-11-03T13:57:28.104 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88845.log.gz 2023-11-03T13:57:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93036.log 2023-11-03T13:57:28.104 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95231.log.gz 2023-11-03T13:57:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91510.log 2023-11-03T13:57:28.105 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90486.log.gz 2023-11-03T13:57:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93941.log 2023-11-03T13:57:28.105 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93036.log.gz 2023-11-03T13:57:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93186.log 2023-11-03T13:57:28.106 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91510.log.gz 2023-11-03T13:57:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84557.log 2023-11-03T13:57:28.106 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93941.log.gz 2023-11-03T13:57:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95903.log 2023-11-03T13:57:28.107 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93186.log.gz 2023-11-03T13:57:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89210.log 2023-11-03T13:57:28.107 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84557.log.gz 2023-11-03T13:57:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90456.log 2023-11-03T13:57:28.108 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95903.log.gz 2023-11-03T13:57:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95928.log 2023-11-03T13:57:28.108 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89210.log.gz 2023-11-03T13:57:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90456.log.gz 2023-11-03T13:57:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94095.log 2023-11-03T13:57:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88869.log 2023-11-03T13:57:28.109 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95928.log.gz 2023-11-03T13:57:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94851.log 2023-11-03T13:57:28.110 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94095.log.gz 2023-11-03T13:57:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88869.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88869.log.gz -5 2023-11-03T13:57:28.111 INFO:teuthology.orchestra.run.smithi073.stderr: --verbose -- /var/log/ceph/ceph-client.admin.94683.log 2023-11-03T13:57:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89621.log 2023-11-03T13:57:28.111 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94851.log.gz 2023-11-03T13:57:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94683.log.gz 2023-11-03T13:57:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96150.log 2023-11-03T13:57:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90274.log 2023-11-03T13:57:28.112 INFO:teuthology.orchestra.run.smithi073.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.89621.log.gz 2023-11-03T13:57:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94384.log 2023-11-03T13:57:28.113 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96150.log.gz 2023-11-03T13:57:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89781.log 2023-11-03T13:57:28.113 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90274.log.gz 2023-11-03T13:57:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90116.log 2023-11-03T13:57:28.114 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94384.log.gz 2023-11-03T13:57:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92865.log 2023-11-03T13:57:28.114 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89781.log.gz 2023-11-03T13:57:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94610.log 2023-11-03T13:57:28.115 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90116.log.gz 2023-11-03T13:57:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92865.log.gz 2023-11-03T13:57:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91874.log 2023-11-03T13:57:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88746.log 2023-11-03T13:57:28.116 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94610.log.gz 2023-11-03T13:57:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91874.log.gz 2023-11-03T13:57:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93207.log 2023-11-03T13:57:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90363.log 2023-11-03T13:57:28.117 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88746.log.gz 2023-11-03T13:57:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90668.log 2023-11-03T13:57:28.117 INFO:teuthology.orchestra.run.smithi073.stderr: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.93207.log.gz 2023-11-03T13:57:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94946.log 2023-11-03T13:57:28.118 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90363.log.gz 2023-11-03T13:57:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96678.log 2023-11-03T13:57:28.118 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90668.log.gz 2023-11-03T13:57:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94946.log.gz 2023-11-03T13:57:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95374.log 2023-11-03T13:57:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89829.log 2023-11-03T13:57:28.120 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96678.log.gz 2023-11-03T13:57:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95374.log.gz 2023-11-03T13:57:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94043.log 2023-11-03T13:57:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93466.log 2023-11-03T13:57:28.121 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89829.log.gz 2023-11-03T13:57:28.121 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91606.log 2023-11-03T13:57:28.121 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94043.log.gz 2023-11-03T13:57:28.121 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.93466.log: /var/log/ceph/ceph-client.xx-profile-rw.95183.log 2023-11-03T13:57:28.122 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93466.log.gz 2023-11-03T13:57:28.122 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91606.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-03T13:57:28.122 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91606.log.gz 2023-11-03T13:57:28.122 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95183.log.gz 2023-11-03T13:57:28.123 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93060.log 2023-11-03T13:57:28.123 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93277.log 2023-11-03T13:57:28.123 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93514.log 2023-11-03T13:57:28.123 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93060.log.gz 2023-11-03T13:57:28.124 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84280.log 2023-11-03T13:57:28.124 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93277.log.gz 2023-11-03T13:57:28.124 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96030.log 2023-11-03T13:57:28.125 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93514.log.gz 2023-11-03T13:57:28.125 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.84280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92535.log 2023-11-03T13:57:28.125 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84280.log.gz 2023-11-03T13:57:28.125 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96030.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.96030.log.gz 2023-11-03T13:57:28.126 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90536.log 2023-11-03T13:57:28.126 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94485.log 2023-11-03T13:57:28.126 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92535.log.gz 2023-11-03T13:57:28.126 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93440.log 2023-11-03T13:57:28.127 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90536.log.gz 2023-11-03T13:57:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94485.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.94485.log.gz /var/log/ceph/ceph-client.admin.92143.log 2023-11-03T13:57:28.127 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-11-03T13:57:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89853.log 2023-11-03T13:57:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93440.log.gz 2023-11-03T13:57:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88648.log 2023-11-03T13:57:28.128 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92143.log.gz 2023-11-03T13:57:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89853.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.89853.log.gz 2023-11-03T13:57:28.129 INFO:teuthology.orchestra.run.smithi073.stderr: /var/log/ceph/ceph-client.admin.96198.log 2023-11-03T13:57:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92042.log 2023-11-03T13:57:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88648.log.gz 2023-11-03T13:57:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96324.log 2023-11-03T13:57:28.130 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96198.log.gz 2023-11-03T13:57:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91801.log 2023-11-03T13:57:28.130 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92042.log.gz 2023-11-03T13:57:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88795.log 2023-11-03T13:57:28.131 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96324.log.gz 2023-11-03T13:57:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91801.log.gz 2023-11-03T13:57:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89668.log 2023-11-03T13:57:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88013.log 2023-11-03T13:57:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88795.log.gz 2023-11-03T13:57:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89668.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.89668.log.gz -5 2023-11-03T13:57:28.132 INFO:teuthology.orchestra.run.smithi073.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90786.log 2023-11-03T13:57:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87673.log 2023-11-03T13:57:28.133 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88013.log.gz 2023-11-03T13:57:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90511.log 0.0% 2023-11-03T13:57:28.133 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90786.log.gz 2023-11-03T13:57:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94145.log 2023-11-03T13:57:28.134 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87673.log.gz 2023-11-03T13:57:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90511.log.gz 2023-11-03T13:57:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88771.log 2023-11-03T13:57:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rw.95255.log 2023-11-03T13:57:28.135 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94145.log.gz 2023-11-03T13:57:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88771.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88771.log.gz 2023-11-03T13:57:28.136 INFO:teuthology.orchestra.run.smithi073.stderr: --verbose -- /var/log/ceph/ceph-client.admin.89247.log 2023-11-03T13:57:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rw.95255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rw.95255.log.gz 2023-11-03T13:57:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91435.log 2023-11-03T13:57:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87593.log 2023-11-03T13:57:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89247.log.gz 2023-11-03T13:57:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91922.log 2023-11-03T13:57:28.137 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91435.log.gz 2023-11-03T13:57:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93086.log 2023-11-03T13:57:28.138 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87593.log.gz 2023-11-03T13:57:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94264.log 2023-11-03T13:57:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91922.log.gz 2023-11-03T13:57:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93086.log.gz 2023-11-03T13:57:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90808.log 2023-11-03T13:57:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89901.log 2023-11-03T13:57:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94264.log.gz 2023-11-03T13:57:28.140 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90808.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90808.log.gz 2023-11-03T13:57:28.140 INFO:teuthology.orchestra.run.smithi073.stderr: --verbose -- /var/log/ceph/ceph-client.admin.94069.log 2023-11-03T13:57:28.140 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89901.log.gz 2023-11-03T13:57:28.140 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91089.log 2023-11-03T13:57:28.141 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94069.log.gz 2023-11-03T13:57:28.141 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96348.log 2023-11-03T13:57:28.141 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87947.log 2023-11-03T13:57:28.141 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91089.log.gz 2023-11-03T13:57:28.142 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96348.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.96348.log.gz 2023-11-03T13:57:28.142 INFO:teuthology.orchestra.run.smithi073.stderr: /var/log/ceph/ceph-client.xx-profile-rd.95609.log 2023-11-03T13:57:28.142 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95877.log 2023-11-03T13:57:28.142 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87947.log.gz 2023-11-03T13:57:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93992.log 2023-11-03T13:57:28.143 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95609.log.gz 2023-11-03T13:57:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96594.log 2023-11-03T13:57:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95877.log.gz 2023-11-03T13:57:28.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93765.log 2023-11-03T13:57:28.144 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93992.log.gz 2023-11-03T13:57:28.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92243.log 2023-11-03T13:57:28.145 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96594.log.gz 2023-11-03T13:57:28.145 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-03T13:57:28.145 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93765.log.gz 2023-11-03T13:57:28.145 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92243.log.gz 2023-11-03T13:57:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91630.log 2023-11-03T13:57:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91068.log 2023-11-03T13:57:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91044.log 2023-11-03T13:57:28.146 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91630.log.gz 2023-11-03T13:57:28.147 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89497.log 2023-11-03T13:57:28.147 INFO:teuthology.orchestra.run.smithi073.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.91068.log.gz 2023-11-03T13:57:28.147 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95777.log 2023-11-03T13:57:28.147 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91044.log.gz 2023-11-03T13:57:28.148 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93841.log 2023-11-03T13:57:28.148 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89497.log.gz 2023-11-03T13:57:28.148 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.95090.log 2023-11-03T13:57:28.148 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95777.log.gz 2023-11-03T13:57:28.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95422.log 2023-11-03T13:57:28.149 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93841.log.gz 2023-11-03T13:57:28.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95090.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95090.log.gz 2023-11-03T13:57:28.150 INFO:teuthology.orchestra.run.smithi073.stderr: -- /var/log/ceph/ceph-client.admin.94310.log 2023-11-03T13:57:28.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95422.log.gz 2023-11-03T13:57:28.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90196.log 2023-11-03T13:57:28.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94310.log.gz 2023-11-03T13:57:28.151 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93331.log 2023-11-03T13:57:28.151 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90196.log.gz 2023-11-03T13:57:28.151 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-03T13:57:28.151 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93331.log.gz 2023-11-03T13:57:28.152 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96460.log 2023-11-03T13:57:28.152 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91560.log 2023-11-03T13:57:28.152 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96460.log.gz 2023-11-03T13:57:28.152 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89713.log 2023-11-03T13:57:28.153 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91560.log.gz 2023-11-03T13:57:28.153 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91215.log 2023-11-03T13:57:28.153 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89713.log.gz 2023-11-03T13:57:28.153 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89993.log 2023-11-03T13:57:28.154 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91215.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.91215.log.gz 2023-11-03T13:57:28.154 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92219.log 2023-11-03T13:57:28.154 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89993.log.gz 2023-11-03T13:57:28.155 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95980.log 2023-11-03T13:57:28.155 INFO:teuthology.orchestra.run.smithi073.stderr: 94.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-03T13:57:28.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92219.log.gz 2023-11-03T13:57:28.155 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96415.log 2023-11-03T13:57:28.156 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.95980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95980.log.gz 2023-11-03T13:57:28.156 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94970.log 2023-11-03T13:57:28.156 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96415.log.gz 2023-11-03T13:57:28.156 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-03T13:57:28.157 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.94970.log: /var/log/ceph/ceph-client.admin.88623.log 2023-11-03T13:57:28.157 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94970.log.gz 2023-11-03T13:57:28.157 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94461.log 2023-11-03T13:57:28.158 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88623.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88623.log.gz 2023-11-03T13:57:28.158 INFO:teuthology.orchestra.run.smithi073.stderr: -- /var/log/ceph/ceph-client.xx-profile-ro.95066.log 2023-11-03T13:57:28.158 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94461.log.gz 2023-11-03T13:57:28.159 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91994.log 2023-11-03T13:57:28.159 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95066.log.gz 2023-11-03T13:57:28.159 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88139.log 2023-11-03T13:57:28.159 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91994.log.gz 2023-11-03T13:57:28.160 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90342.log 2023-11-03T13:57:28.160 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88139.log.gz 2023-11-03T13:57:28.161 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90692.log 2023-11-03T13:57:28.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90342.log.gz 2023-11-03T13:57:28.161 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95470.log 2023-11-03T13:57:28.161 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90692.log.gz 2023-11-03T13:57:28.162 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88033.log 2023-11-03T13:57:28.162 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95470.log.gz 2023-11-03T13:57:28.162 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91755.log 2023-11-03T13:57:28.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88033.log.gz 2023-11-03T13:57:28.163 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-03T13:57:28.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91755.log.gz 2023-11-03T13:57:28.163 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90738.log 2023-11-03T13:57:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92490.log 2023-11-03T13:57:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90738.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92769.log 2023-11-03T13:57:28.164 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90738.log.gz 2023-11-03T13:57:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92490.log.gz 2023-11-03T13:57:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93138.log 2023-11-03T13:57:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90830.log 2023-11-03T13:57:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92769.log.gz 2023-11-03T13:57:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93138.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94242.log 2023-11-03T13:57:28.166 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-11-03T13:57:28.166 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89136.log 2023-11-03T13:57:28.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90830.log: /var/log/ceph/ceph-client.admin.94242.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90830.log.gz 2023-11-03T13:57:28.167 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94242.log.gz 2023-11-03T13:57:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89269.log 2023-11-03T13:57:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92961.log 2023-11-03T13:57:28.167 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89136.log.gz 2023-11-03T13:57:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89269.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87675.log 2023-11-03T13:57:28.168 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-11-03T13:57:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91411.log 2023-11-03T13:57:28.168 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92961.log: /var/log/ceph/ceph-client.admin.87675.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92961.log.gz 2023-11-03T13:57:28.169 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87675.log.gz 2023-11-03T13:57:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88550.log 2023-11-03T13:57:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89877.log 2023-11-03T13:57:28.169 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91411.log.gz 2023-11-03T13:57:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88550.log.gz 2023-11-03T13:57:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89971.log 2023-11-03T13:57:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95704.log 2023-11-03T13:57:28.170 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.89971.log: /var/log/ceph/ceph-client.admin.89877.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89971.log.gz 2023-11-03T13:57:28.171 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89877.log.gz 2023-11-03T13:57:28.171 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96222.log 2023-11-03T13:57:28.171 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90620.log 2023-11-03T13:57:28.171 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-rd.95704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95704.log.gz 2023-11-03T13:57:28.172 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96222.log.gz 2023-11-03T13:57:28.172 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90716.log 2023-11-03T13:57:28.172 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.90620.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.90620.log.gz 2023-11-03T13:57:28.173 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.96054.log 2023-11-03T13:57:28.173 INFO:teuthology.orchestra.run.smithi073.stderr:gzip/var/log/ceph/ceph-client.admin.90716.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90716.log.gz 2023-11-03T13:57:28.173 INFO:teuthology.orchestra.run.smithi073.stderr: /var/log/ceph/ceph-client.xx-profile-ro.95136.log 2023-11-03T13:57:28.173 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94634.log 2023-11-03T13:57:28.174 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96054.log.gz 2023-11-03T13:57:28.174 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.xx-profile-ro.95136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.95136.log.gz 2023-11-03T13:57:28.174 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88475.log 2023-11-03T13:57:28.174 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91289.log 2023-11-03T13:57:28.175 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.94634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94634.log.gz 2023-11-03T13:57:28.175 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88475.log.gzgzip 2023-11-03T13:57:28.175 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91116.log 2023-11-03T13:57:28.175 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92466.log 2023-11-03T13:57:28.176 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.91116.log: /var/log/ceph/ceph-client.admin.91289.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91116.log.gz 2023-11-03T13:57:28.176 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91289.log.gz 2023-11-03T13:57:28.176 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-ro.94994.log 2023-11-03T13:57:28.176 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.92466.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.92466.log.gz 2023-11-03T13:57:28.177 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.88063.log 2023-11-03T13:57:28.177 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96720.log 2023-11-03T13:57:28.177 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88063.log: /var/log/ceph/ceph-client.xx-profile-ro.94994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88063.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.xx-profile-ro.94994.log.gz 2023-11-03T13:57:28.177 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-11-03T13:57:28.178 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88697.log 2023-11-03T13:57:28.178 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.xx-profile-rd.95632.log 2023-11-03T13:57:28.178 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.96720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96720.log.gz 2023-11-03T13:57:28.179 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.88697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88697.log.gzgzip 2023-11-03T13:57:28.179 INFO:teuthology.orchestra.run.smithi073.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.93916.log 2023-11-03T13:57:28.179 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93661.log 2023-11-03T13:57:28.179 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93916.log: /var/log/ceph/ceph-client.xx-profile-rd.95632.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93916.log.gz 2023-11-03T13:57:28.180 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.xx-profile-rd.95632.log.gz 2023-11-03T13:57:28.180 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-03T13:57:28.180 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93661.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93253.log 2023-11-03T13:57:28.180 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93661.log.gz 2023-11-03T13:57:28.181 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87907.log 2023-11-03T13:57:28.181 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.93253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93253.log.gz 2023-11-03T13:57:28.181 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.87907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87907.log.gz 2023-11-03T13:57:28.182 INFO:teuthology.orchestra.run.smithi073.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-03T13:57:28.182 INFO:teuthology.orchestra.run.smithi073.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-03T13:57:28.215 INFO:teuthology.orchestra.run.smithi073.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-03T13:57:28.220 INFO:teuthology.orchestra.run.smithi073.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-03T13:57:28.280 INFO:teuthology.orchestra.run.smithi073.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-03T13:57:28.291 INFO:teuthology.orchestra.run.smithi073.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-03T13:57:28.681 INFO:teuthology.orchestra.run.smithi073.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-03T13:57:28.683 INFO:teuthology.orchestra.run.smithi073.stderr: 2023-11-03T13:57:28.684 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m0.831s 2023-11-03T13:57:28.684 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m2.415s 2023-11-03T13:57:28.685 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.215s 2023-11-03T13:57:28.686 INFO:tasks.ceph:Archiving logs... 2023-11-03T13:57:28.687 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/remote/smithi073/log 2023-11-03T13:57:28.688 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-03T13:57:29.426 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-03T13:57:29.445 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-03T13:57:29.446 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-03T13:57:29.500 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-03T13:57:29.510 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-03T13:57:29.511 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-03T13:57:29.528 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2023-11-03T13:57:29.532 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-03T13:57:29.532 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-11-03T13:57:29.532 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +2367us[+2102us] +/- 103ms 2023-11-03T13:57:29.533 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -1477us[-1477us] +/- 49ms 2023-11-03T13:57:29.533 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv03.front.sepia.ceph.com 3 6 377 23 +436us[ +170us] +/- 78ms 2023-11-03T13:57:29.533 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-03T13:57:29.535 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-03T13:57:29.549 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-03T13:57:29.550 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-03T13:57:29.563 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-03T13:57:29.666 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-03T13:57:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:57:29.700 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-03T13:57:29.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-03T13:57:29.726 INFO:teuthology.task.internal:Duration was 952.275114 seconds 2023-11-03T13:57:29.727 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-03T13:57:29.736 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-03T13:57:29.737 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-03T13:57:29.783 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-03T13:57:29.957 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-03T13:57:29.958 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2023-11-03T13:57:29.958 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-03T13:57:30.018 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-03T13:57:30.018 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-03T13:57:30.086 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-03T13:57:30.087 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-03T13:57:30.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-03T13:57:30.232 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-03T13:57:30.232 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-03T13:57:30.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-03T13:57:30.272 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-03T13:57:30.325 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2023-11-03T13:57:30.345 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-03T13:57:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:57:30.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-03T13:57:30.409 INFO:teuthology.task.internal:Transferring archived files... 2023-11-03T13:57:30.410 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381/remote/smithi073 2023-11-03T13:57:30.410 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-03T13:57:30.475 INFO:teuthology.task.internal:Removing archive directory... 2023-11-03T13:57:30.475 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-03T13:57:30.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-03T13:57:30.534 INFO:teuthology.task.internal:Not uploading archives. 2023-11-03T13:57:30.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-03T13:57:30.544 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-03T13:57:30.545 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-03T13:57:30.578 INFO:teuthology.orchestra.run.smithi073.stdout: 265960 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 3 13:57 /home/ubuntu/cephtest 2023-11-03T13:57:30.579 INFO:teuthology.orchestra.run.smithi073.stdout: 402848 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 3 13:50 /home/ubuntu/cephtest/mnt.0 2023-11-03T13:57:30.579 INFO:teuthology.orchestra.run.smithi073.stdout: 402849 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 3 13:51 /home/ubuntu/cephtest/mnt.0/client.0 2023-11-03T13:57:30.579 INFO:teuthology.orchestra.run.smithi073.stdout: 416757 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 3 13:54 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-03T13:57:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-03T13:57:30.581 INFO:teuthology.orchestra.run.smithi073.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-03T13:57:30.582 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi073 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-03T13:57:30.588 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-03T13:57:30.598 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-03T13:57:30.607 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi073 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-03T13:57:30.608 INFO:teuthology.nuke:Checking targets against current locks 2023-11-03T13:57:30.637 DEBUG:teuthology.nuke:shortname: smithi073 2023-11-03T13:57:30.638 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-03T13:57:30.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sseshasa-2023-11-03_06:54:32-rados-wip-sseshasa-testing-2023-11-02-1957-distro-default-smithi/7445381', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-03 13:33:39.511881', 'locked_by': 'scheduled_sseshasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWvSLCWudOvyAAkiWxaMd3EWqD2QOGdrg1rRjM3vPXSLunrPnXTJl/QHegXRGFOvdmGgnOtsiCgWjLBVxOhgcE='} 2023-11-03T13:57:30.719 INFO:teuthology.orchestra.console.smithi073:Power off 2023-11-03T13:57:30.721 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-03T13:57:30.751 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2023-11-03T13:57:30.751 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-03T13:57:30.777 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-11-03T13:57:34.778 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-03T13:57:34.889 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-11-03T13:57:38.890 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-03T13:57:39.002 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2023-11-03T13:57:43.006 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-03T13:57:43.116 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2023-11-03T13:57:43.218 INFO:teuthology.orchestra.console.smithi073:Power off completed 2023-11-03T13:57:43.327 INFO:teuthology.run:Summary data: description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} duration: 952.2751135826111 failure_reason: 'Command failed (workunit test cephtool/test.sh) on smithi073 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=10145169af62d2c41c01dd8795eeddb1b869c21a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh''' flavor: default owner: scheduled_sseshasa@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6475c9c06d9f463bafdf41f2a91ce6ab status: fail success: false 2023-11-03T13:57:43.446 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-03T13:57:43.526 INFO:teuthology.run:FAIL