2024-05-09T02:49:53.304 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T02:49:53.305 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi --archive /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454 --owner scheduled_sjust@teuthology --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} -- /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/orig.config.yaml 2024-05-09T02:49:53.355 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T02:49:53.439 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454 branch: wip-sjust-balanced-read-testing-2024-05-08 description: rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} email: null first_in_suite: false job_id: '7699454' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 - volumes: size: 50 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sjust-balanced-read-testing-2024-05-08 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd inject bad map crc probability: 0.1 osd max backfills: 9 osd max markdown count: 1000 osd max scrubs: 1 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - must scrub before tier agent can activate - \(POOL_APP_NOT_ENABLED\) sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sjust-balanced-read-testing-2024-05-08 sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 owner: scheduled_sjust@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 4494 sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 sleep_before_teardown: 0 subset: 10/3000 suite: rados:thrash suite_branch: wip-sjust-balanced-read-testing-2024-05-08 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAz2REg4lMucipWe/Qym4OExMaGQThHxYzzcoO5SnYyhhtPFTUBV1VOFUK4Im5i5selAYjBOh5T04h7agwIORdE= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEEG9lnjv2iuYW/yBzKLhScFVmU519RnxmwnqXsoMDNaGITzCPfZU3OoyDf6nQ2Q4iPsTD2D9CbR6lllHxsPs0= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWZvl6DWaFvDjAubG4RwJqIdDPPiqZ7+JXnpDETpJZ+YnyXfJCg9qp3bhHt2WlZCt7MkzgHzvZKEwQWtRqRi5A= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY4lZiaVWRxSjv4ZaODKGlHFKK4EnLuUQ37HMQtVXu99F/L6Kjnnbd5S8OdOHhcGHfZDVr05ctdvD6JJ63RnYw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - ceph config set mgr mgr/balancer/log_level debug - ceph balancer status - ceph osd set-require-min-compat-client reef - ceph balancer mode upmap-read - ceph balancer on - ceph balancer status - ceph balancer status detail - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 min_in: 2 timeout: 1200 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - rados: balance_reads: true clients: - client.0 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-09_02:35:04 tube: smithi user: sjust verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T02:49:53.439 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa; will attempt to use it 2024-05-09T02:49:53.440 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks 2024-05-09T02:49:53.440 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T02:49:53.442 INFO:teuthology.task.internal:Checking packages... 2024-05-09T02:49:53.465 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c152e9f40d11b517c5526b6e8144d52000d7e690' 2024-05-09T02:49:53.466 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T02:49:53.466 INFO:teuthology.packaging:ref: None 2024-05-09T02:49:53.466 INFO:teuthology.packaging:tag: None 2024-05-09T02:49:53.466 INFO:teuthology.packaging:branch: wip-sjust-balanced-read-testing-2024-05-08 2024-05-09T02:49:53.466 INFO:teuthology.packaging:sha1: c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T02:49:53.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-sjust-balanced-read-testing-2024-05-08 2024-05-09T02:49:53.608 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3603.gc152e9f4 2024-05-09T02:49:53.609 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T02:49:53.617 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T02:49:53.617 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T02:49:53.625 INFO:teuthology.task.internal:Saving configuration 2024-05-09T02:49:53.643 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T02:49:53.650 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T02:49:53.671 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.130185', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAz2REg4lMucipWe/Qym4OExMaGQThHxYzzcoO5SnYyhhtPFTUBV1VOFUK4Im5i5selAYjBOh5T04h7agwIORdE='} 2024-05-09T02:49:53.692 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.128609', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEEG9lnjv2iuYW/yBzKLhScFVmU519RnxmwnqXsoMDNaGITzCPfZU3OoyDf6nQ2Q4iPsTD2D9CbR6lllHxsPs0='} 2024-05-09T02:49:53.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.131556', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWZvl6DWaFvDjAubG4RwJqIdDPPiqZ7+JXnpDETpJZ+YnyXfJCg9qp3bhHt2WlZCt7MkzgHzvZKEwQWtRqRi5A='} 2024-05-09T02:49:53.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.132933', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY4lZiaVWRxSjv4ZaODKGlHFKK4EnLuUQ37HMQtVXu99F/L6Kjnnbd5S8OdOHhcGHfZDVr05ctdvD6JJ63RnYw='} 2024-05-09T02:49:53.731 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T02:49:53.739 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-05-09T02:49:53.740 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-05-09T02:49:53.740 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-05-09T02:49:53.740 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-05-09T02:49:53.740 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T02:49:53.874 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f944fe65ee0>, signals=[15]) 2024-05-09T02:49:53.875 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T02:49:53.882 INFO:teuthology.task.internal:Opening connections... 2024-05-09T02:49:53.882 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T02:49:53.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:49:53.981 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T02:49:53.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:49:54.053 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T02:49:54.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:49:54.127 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T02:49:54.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:49:54.199 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T02:49:54.211 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-05-09T02:49:54.229 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-05-09T02:49:54.230 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-05-09T02:49:54.286 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-05-09T02:49:54.286 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-05-09T02:49:54.286 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-05-09T02:49:54.286 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:49:54.287 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:49:54.288 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-05-09T02:49:54.310 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-05-09T02:49:54.328 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-05-09T02:49:54.328 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-05-09T02:49:54.386 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:49:54.387 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:49:54.388 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-05-09T02:49:54.410 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-05-09T02:49:54.430 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-05-09T02:49:54.430 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-05-09T02:49:54.487 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:49:54.488 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:49:54.488 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:49:54.488 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:49:54.488 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:49:54.488 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:49:54.488 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-05-09T02:49:54.510 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-05-09T02:49:54.528 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-05-09T02:49:54.528 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-05-09T02:49:54.585 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-05-09T02:49:54.585 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-05-09T02:49:54.585 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-05-09T02:49:54.585 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:49:54.585 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:49:54.586 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:49:54.586 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-05-09T02:49:54.602 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T02:49:54.612 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T02:49:54.624 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T02:49:54.625 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:49:54.627 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:49:54.630 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:49:54.634 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:49:54.650 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T02:49:54.661 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T02:49:54.661 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:49:54.685 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:49:54.687 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:49:54.689 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:49:54.798 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T02:49:54.894 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T02:49:54.914 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-05-09T02:49:54.914 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-05-09T02:49:54.914 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:49:54.915 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:49:54.915 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:49:54.916 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:49:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:49:54.932 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-05-09T02:49:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:49:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:49:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:49:54.935 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-05-09T02:49:54.935 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-05-09T02:49:54.935 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-05-09T02:49:54.990 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:49:54.990 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-437.el9.x86_64 2024-05-09T02:49:54.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-05-09T02:49:54.991 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:49:54.991 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-437.el9.x86_64 2024-05-09T02:49:54.992 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-05-09T02:49:54.993 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:49:54.993 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-437.el9.x86_64 2024-05-09T02:49:54.993 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-05-09T02:49:54.994 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:49:54.994 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-437.el9.x86_64 2024-05-09T02:49:54.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-05-09T02:49:58.107 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-09T02:49:58.423 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.1 MB 00:01 2024-05-09T02:49:58.690 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.1 MB 00:01 2024-05-09T02:49:58.927 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.1 MB 00:02 2024-05-09T02:50:00.018 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-09T02:50:01.707 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-05-09T02:50:02.175 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 19 MB 00:02 2024-05-09T02:50:03.078 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-05-09T02:50:03.953 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.2 MB 00:00 2024-05-09T02:50:05.580 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-05-09T02:50:06.641 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-05-09T02:50:06.892 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-05-09T02:50:07.162 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-09T02:50:07.366 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-05-09T02:50:08.400 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-05-09T02:50:09.259 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-05-09T02:50:09.846 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-05-09T02:50:09.912 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-09T02:50:10.973 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-05-09T02:50:11.969 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-05-09T02:50:13.079 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-05-09T02:50:13.976 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:50:13.976 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:50:14.114 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T02:50:14.139 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-05-09T02:50:14.140 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:50:14.141 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 78 M 2024-05-09T02:50:14.141 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-05-09T02:50:14.141 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T02:50:14.521 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 1.8 MB/s | 484 kB 00:00 2024-05-09T02:50:15.160 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-05-09T02:50:15.805 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 10 MB/s | 15 MB 00:01 2024-05-09T02:50:16.172 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-09T02:50:16.606 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 13 MB/s | 28 MB 00:02 2024-05-09T02:50:17.148 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-05-09T02:50:17.148 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:50:17.148 INFO:teuthology.orchestra.run.smithi072.stdout:Total 26 MB/s | 78 MB 00:03 2024-05-09T02:50:17.157 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:50:17.157 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:50:17.252 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:50:17.258 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-05-09T02:50:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:50:17.260 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 78 M 2024-05-09T02:50:17.260 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 123 M 2024-05-09T02:50:17.260 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T02:50:17.419 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 5.1 MB/s | 484 kB 00:00 2024-05-09T02:50:17.829 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T02:50:17.920 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-05-09T02:50:18.174 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T02:50:18.174 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T02:50:18.195 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:50:18.195 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:50:18.200 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T02:50:18.301 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T02:50:18.313 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T02:50:18.314 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:50:18.315 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-05-09T02:50:18.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:50:18.315 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 78 M 2024-05-09T02:50:18.315 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-05-09T02:50:18.315 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T02:50:18.470 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 30 MB/s | 34 MB 00:01 2024-05-09T02:50:18.753 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 21 MB/s | 28 MB 00:01 2024-05-09T02:50:18.753 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:50:18.754 INFO:teuthology.orchestra.run.smithi181.stdout:Total 52 MB/s | 78 MB 00:01 2024-05-09T02:50:19.096 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T02:50:19.097 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T02:50:19.382 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 521 kB/s | 484 kB 00:00 2024-05-09T02:50:19.436 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T02:50:19.781 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T02:50:19.781 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T02:50:20.016 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 9.9 MB/s | 15 MB 00:01 2024-05-09T02:50:20.240 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:50:20.240 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:50:20.341 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T02:50:20.352 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:50:20.353 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-05-09T02:50:20.354 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:50:20.354 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 78 M 2024-05-09T02:50:20.354 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 123 M 2024-05-09T02:50:20.354 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T02:50:20.641 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-05-09T02:50:20.668 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.7 MB/s | 484 kB 00:00 2024-05-09T02:50:20.727 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T02:50:20.728 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T02:50:21.513 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T02:50:21.699 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 12 MB/s | 28 MB 00:02 2024-05-09T02:50:21.700 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:50:21.700 INFO:teuthology.orchestra.run.smithi195.stdout:Total 23 MB/s | 78 MB 00:03 2024-05-09T02:50:21.785 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-05-09T02:50:22.186 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:50:22.295 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:22.382 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T02:50:22.402 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 16 MB/s | 28 MB 00:01 2024-05-09T02:50:22.736 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-05-09T02:50:22.736 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:50:22.736 INFO:teuthology.orchestra.run.smithi096.stdout:Total 33 MB/s | 78 MB 00:02 2024-05-09T02:50:22.749 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T02:50:22.749 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T02:50:23.136 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T02:50:23.417 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T02:50:23.694 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T02:50:23.695 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T02:50:23.767 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T02:50:23.768 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T02:50:23.833 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:50:23.911 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:24.696 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T02:50:24.696 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T02:50:24.939 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:25.152 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:26.130 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T02:50:26.576 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:26.730 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:26.735 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:50:26.816 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:27.097 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T02:50:27.742 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:50:27.827 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:29.511 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:29.645 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:29.875 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:29.926 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:30.489 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:50:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:31.484 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:31.549 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:34.363 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:34.414 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:34.699 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:35.403 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:50:35.513 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:36.385 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:39.132 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:50:40.261 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:05.990 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:06.215 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:06.215 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:51:06.215 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:51:06.215 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:51:06.354 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:06.354 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:51:06.355 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:51:06.355 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:51:07.038 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:07.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:51:07.038 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T02:51:07.038 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.039 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.039 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.039 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.039 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:51:07.039 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T02:51:07.376 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.419 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.420 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:51:07.420 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T02:51:07.446 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.447 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.447 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.447 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:51:07.447 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:51:07.447 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-05-09T02:51:07.447 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:51:07.448 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:07.497 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.497 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-05-09T02:51:07.531 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-05-09T02:51:07.531 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-05-09T02:51:07.531 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-05-09T02:51:07.531 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 68.el9 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8264254 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-05-09T02:51:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:51:07.533 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:51:07.534 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:51:07.534 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:51:07.863 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:07.960 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:07.961 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.961 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:51:07.961 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:51:07.961 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:51:07.961 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-09T02:51:07.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:51:07.961 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:08.012 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:08.012 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:08.012 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-05-09T02:51:08.042 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:51:08.043 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:51:08.044 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:51:08.045 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:51:08.045 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:51:08.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:51:08.126 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-05-09T02:51:08.313 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:08.353 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:08.695 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:08.695 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:51:08.695 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:51:08.695 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:51:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:51:09.488 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T02:51:09.531 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:51:09.562 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-05-09T02:51:09.564 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-05-09T02:51:09.580 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.2owFYLPk5J 2024-05-09T02:51:09.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2owFYLPk5J 2024-05-09T02:51:09.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.2owFYLPk5J 2024-05-09T02:51:09.782 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.2owFYLPk5J is 6KB 2024-05-09T02:51:09.831 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.2owFYLPk5J 2024-05-09T02:51:09.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:51:09.847 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:09.906 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:51:09.906 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:51:09.906 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:51:09.907 INFO:teuthology.orchestra.run.smithi181.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:51:09.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.917 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:09.918 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:51:09.918 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:51:09.918 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:51:09.918 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-05-09T02:51:09.918 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:51:09.919 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:09.969 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:09.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:09.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:51:10.009 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-05-09T02:51:10.010 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:51:10.011 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:51:10.012 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:51:10.012 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:51:10.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:51:10.046 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:10.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-05-09T02:51:10.084 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:10.304 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:51:10.327 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-05-09T02:51:10.329 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-05-09T02:51:10.356 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.2ZELvWBQnq 2024-05-09T02:51:10.356 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2ZELvWBQnq 2024-05-09T02:51:10.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.2ZELvWBQnq 2024-05-09T02:51:10.443 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:10.443 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:51:10.443 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:51:10.443 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:51:10.560 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.2ZELvWBQnq is 6KB 2024-05-09T02:51:10.580 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.2ZELvWBQnq 2024-05-09T02:51:10.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:51:10.659 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:51:10.659 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:51:10.659 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:51:10.659 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:51:10.660 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:51:10.696 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-05-09T02:51:10.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:51:11.242 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T02:51:11.585 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:11.662 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:11.662 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:51:11.662 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:51:11.663 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:51:11.663 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-09T02:51:11.663 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:51:11.663 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:51:11.712 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:51:11.713 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:51:11.713 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-05-09T02:51:11.744 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-05-09T02:51:11.745 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:51:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:51:11.747 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:51:11.747 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:51:11.747 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:51:12.125 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:51:12.169 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-05-09T02:51:12.171 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-05-09T02:51:12.227 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.INfQArDSZi 2024-05-09T02:51:12.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.INfQArDSZi 2024-05-09T02:51:12.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.INfQArDSZi 2024-05-09T02:51:12.432 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-05-09T02:51:12.442 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.INfQArDSZi is 6KB 2024-05-09T02:51:12.460 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.INfQArDSZi 2024-05-09T02:51:12.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:51:12.536 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:51:12.536 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:51:12.536 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:51:12.536 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:51:12.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:51:12.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-05-09T02:51:13.778 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:51:13.813 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-05-09T02:51:13.820 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-05-09T02:51:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.kVmypvRwDD 2024-05-09T02:51:13.835 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kVmypvRwDD 2024-05-09T02:51:13.910 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.kVmypvRwDD 2024-05-09T02:51:14.042 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.kVmypvRwDD is 6KB 2024-05-09T02:51:14.061 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.kVmypvRwDD 2024-05-09T02:51:14.076 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:51:14.137 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:51:14.137 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:51:14.137 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:51:14.137 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:51:14.138 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:51:14.298 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-05-09T02:51:40.094 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-09T02:51:40.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:51:40.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:51:40.812 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-09T02:51:40.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:51:40.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:51:42.690 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-05-09T02:51:42.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:51:42.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:51:44.302 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-09T02:51:44.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:51:44.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:51:58.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-05-09T02:51:59.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-05-09T02:52:01.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-05-09T02:52:02.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-05-09T02:52:07.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:52:07.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:08.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:52:08.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:10.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:52:10.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:10.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-05-09T02:52:11.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-05-09T02:52:11.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:52:11.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:13.250 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-05-09T02:52:14.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-05-09T02:52:22.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:52:22.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:23.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:52:23.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:25.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:52:25.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:26.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-05-09T02:52:26.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:52:26.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:41.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:52:41.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:52:44.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-05-09T02:52:54.555 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-05-09T02:52:54.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:52:54.951 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:52:54.952 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-05-09T02:52:54.966 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:52:54.966 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:52:54.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:52:54.967 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:52:55.968 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-09T02:52:55.968 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:52:56.066 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-05-09T02:52:56.097 DEBUG:teuthology.parallel:result is None 2024-05-09T02:52:57.232 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-05-09T02:52:57.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:52:57.614 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:52:57.614 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-05-09T02:52:57.628 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:52:57.629 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:52:57.629 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:52:57.629 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:52:58.630 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-05-09T02:52:58.630 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:52:58.691 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-05-09T02:52:58.732 DEBUG:teuthology.parallel:result is None 2024-05-09T02:52:58.855 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-05-09T02:52:59.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:52:59.361 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:52:59.361 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-05-09T02:52:59.386 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:52:59.386 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:52:59.386 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:52:59.386 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:53:00.386 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-09T02:53:00.387 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:53:00.445 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-05-09T02:53:00.547 DEBUG:teuthology.parallel:result is None 2024-05-09T02:53:02.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:53:02.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:53:05.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-05-09T02:53:26.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:53:26.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:54:26.419 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T02:54:50.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:54:50.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:54:50.834 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-05-09T02:54:51.254 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:54:51.254 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:54:51.255 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-05-09T02:54:51.272 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:54:51.273 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:54:51.273 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:54:51.273 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:54:52.273 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-09T02:54:52.274 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:54:52.357 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2024-05-09T02:54:52.393 DEBUG:teuthology.parallel:result is None 2024-05-09T02:54:52.393 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T02:54:52.400 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T02:54:52.400 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:54:52.403 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:54:52.406 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:54:52.408 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:54:52.429 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T02:54:52.435 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T02:54:52.442 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T02:54:52.442 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:54:52.462 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:54:52.465 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:54:52.469 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:54:52.652 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T02:54:52.659 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T02:54:52.659 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:54:52.662 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:54:52.664 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:54:52.666 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:54:52.699 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.706 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.710 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.717 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.726 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.728 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.738 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.740 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:54:52.741 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T02:54:52.748 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T02:54:52.748 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:54:52.755 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:54:52.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:54:52.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:54:52.790 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T02:54:52.801 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T02:54:52.801 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:54:52.823 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:54:52.826 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:54:52.828 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:54:52.846 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:52.923 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:52.977 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.064 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.104 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T02:54:53.104 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:54:53.170 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.298 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.387 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T02:54:53.388 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:54:53.456 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.557 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.683 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T02:54:53.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:54:53.749 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:54:53.854 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:54:53.983 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T02:54:53.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:54:54.050 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-05-09T02:54:54.053 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-05-09T02:54:54.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-05-09T02:54:54.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-05-09T02:54:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:54:54.102 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:54:54.125 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:54:54.137 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:54:54.587 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T02:54:54.594 INFO:teuthology.task.internal:Starting timer... 2024-05-09T02:54:54.594 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T02:54:54.620 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T02:54:54.642 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-09T02:54:54.642 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-05-09T02:54:54.749 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-05-09T02:54:54.751 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-05-09T02:54:54.840 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-05-09T02:54:54.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-05-09T02:54:54.925 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-05-09T02:54:54.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-05-09T02:54:55.004 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-05-09T02:54:55.007 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T02:54:55.007 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-05-09T02:54:55.055 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-05-09T02:54:55.056 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-05-09T02:54:55.083 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-05-09T02:54:55.083 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-05-09T02:54:55.150 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-05-09T02:54:55.151 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-05-09T02:54:55.188 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-05-09T02:54:55.189 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-05-09T02:54:55.189 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:54:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:54:55.220 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:54:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:54:55.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:54:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:54:55.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:54:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:54:55.308 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T02:54:55.308 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:54:55.342 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:54:55.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:54:55.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:54:55.439 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T02:54:55.447 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T02:54:55.467 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T02:54:55.469 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi096.front.sepia.ceph.com,smithi181.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T02:59:03.093 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-05-09T02:59:03.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:59:03.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:03.174 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-05-09T02:59:03.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-05-09T02:59:03.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:59:03.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:03.332 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-05-09T02:59:03.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-09T02:59:03.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:59:03.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:03.495 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-05-09T02:59:03.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-09T02:59:03.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:59:03.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:03.649 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-05-09T02:59:03.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-09T02:59:03.740 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T02:59:03.750 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T02:59:03.751 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T02:59:03.751 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T02:59:03.754 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T02:59:03.754 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T02:59:03.756 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T02:59:03.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T02:59:03.758 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T02:59:03.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T02:59:03.782 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T02:59:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T02:59:03.788 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T02:59:03.796 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T02:59:03.798 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T02:59:03.803 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T02:59:03.811 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T02:59:03.821 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-05-09T02:59:03.823 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-05-09T02:59:03.826 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T02:59:03.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-05-09T02:59:03.832 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-05-09T02:59:03.834 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-05-09T02:59:03.841 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-05-09T02:59:03.847 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T02:59:03.848 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T02:59:03.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-05-09T02:59:03.855 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T02:59:03.860 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T02:59:03.862 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-05-09T02:59:03.862 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T02:59:03.869 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T02:59:03.876 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T02:59:03.890 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T02:59:03.914 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-05-09T02:59:03.917 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-05-09T02:59:03.921 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T02:59:03.922 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-05-09T02:59:03.922 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.922 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.922 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.922 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.923 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.927 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.947 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.951 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T02:59:03.952 INFO:teuthology.run_tasks:Running task install... 2024-05-09T02:59:03.960 DEBUG:teuthology.task.install:project ceph 2024-05-09T02:59:03.961 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690'}} 2024-05-09T02:59:03.961 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690'} 2024-05-09T02:59:03.961 INFO:teuthology.task.install:Using flavor: default 2024-05-09T02:59:03.966 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T02:59:03.966 INFO:teuthology.task.install:extra packages: [] 2024-05-09T02:59:03.967 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'tag': None, 'wait_for_package': False} 2024-05-09T02:59:03.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T02:59:03.969 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'tag': None, 'wait_for_package': False} 2024-05-09T02:59:03.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T02:59:03.970 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'tag': None, 'wait_for_package': False} 2024-05-09T02:59:03.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T02:59:03.972 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'tag': None, 'wait_for_package': False} 2024-05-09T02:59:03.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T02:59:04.201 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/ 2024-05-09T02:59:04.201 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3603.gc152e9f4 2024-05-09T02:59:04.207 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/ 2024-05-09T02:59:04.208 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3603.gc152e9f4 2024-05-09T02:59:04.211 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/ 2024-05-09T02:59:04.211 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3603.gc152e9f4 2024-05-09T02:59:04.215 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/ 2024-05-09T02:59:04.216 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3603.gc152e9f4 2024-05-09T02:59:04.323 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T02:59:04.323 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T02:59:04.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T02:59:04.328 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T02:59:04.328 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T02:59:04.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T02:59:04.349 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-09T02:59:04.349 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c152e9f40d11b517c5526b6e8144d52000d7e690/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T02:59:04.355 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-09T02:59:04.355 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c152e9f40d11b517c5526b6e8144d52000d7e690/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T02:59:04.379 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T02:59:04.380 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T02:59:04.380 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T02:59:04.395 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-sjust-balanced-read-testing-2024-05-08/c152e9f40d11b517c5526b6e8144d52000d7e690/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T02:59:04.395 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T02:59:04.395 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T02:59:04.408 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-09T02:59:04.408 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c152e9f40d11b517c5526b6e8144d52000d7e690/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T02:59:04.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T02:59:04.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T02:59:04.422 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-09T02:59:04.423 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c152e9f40d11b517c5526b6e8144d52000d7e690/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T02:59:04.476 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T02:59:04.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T02:59:04.490 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T02:59:04.496 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T02:59:04.552 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T02:59:04.554 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-05-09T02:59:04.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-05-09T02:59:04.560 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-05-09T02:59:04.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-05-09T02:59:04.563 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T02:59:04.617 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-05-09T02:59:04.619 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-05-09T02:59:04.627 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-05-09T02:59:04.629 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-05-09T02:59:04.788 INFO:teuthology.orchestra.run.smithi195.stdout:54 files removed 2024-05-09T02:59:04.806 INFO:teuthology.orchestra.run.smithi181.stdout:54 files removed 2024-05-09T02:59:04.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-05-09T02:59:04.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-05-09T02:59:04.849 INFO:teuthology.orchestra.run.smithi072.stdout:54 files removed 2024-05-09T02:59:04.864 INFO:teuthology.orchestra.run.smithi096.stdout:54 files removed 2024-05-09T02:59:04.876 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw 2024-05-09T02:59:04.890 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-05-09T02:59:05.415 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 271 kB/s | 84 kB 00:00 2024-05-09T02:59:05.476 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 257 kB/s | 84 kB 00:00 2024-05-09T02:59:05.495 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 253 kB/s | 84 kB 00:00 2024-05-09T02:59:05.495 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 208 kB/s | 84 kB 00:00 2024-05-09T02:59:05.774 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 38 kB/s | 12 kB 00:00 2024-05-09T02:59:05.861 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-05-09T02:59:05.879 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-05-09T02:59:05.920 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-09T02:59:06.124 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-05-09T02:59:06.237 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-05-09T02:59:06.296 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-05-09T02:59:06.321 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-05-09T02:59:06.806 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-09T02:59:07.854 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.1 MB 00:01 2024-05-09T02:59:07.879 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.1 MB 00:01 2024-05-09T02:59:07.999 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-09T02:59:10.588 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-09T02:59:11.181 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-05-09T02:59:12.375 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-05-09T02:59:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-05-09T02:59:15.940 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.2 MB 00:01 2024-05-09T02:59:17.615 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-05-09T02:59:17.801 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.2 MB 00:01 2024-05-09T02:59:18.295 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-09T02:59:18.686 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-05-09T02:59:19.890 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-05-09T02:59:20.252 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-05-09T02:59:21.653 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-09T02:59:22.082 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-05-09T02:59:22.313 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 772 kB/s | 6.2 MB 00:08 2024-05-09T02:59:23.859 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-09T02:59:24.338 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-05-09T02:59:26.225 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T02:59:26.238 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:26.238 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T02:59:26.238 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:26.238 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 11 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.4 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 21 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 25 k 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T02:59:26.239 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 747 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 501 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 45 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 135 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 161 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T02:59:26.240 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 320 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 300 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 100 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout:Install 30 Packages 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-05-09T02:59:26.241 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:26.242 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 60 M 2024-05-09T02:59:26.242 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T02:59:26.812 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-05-09T02:59:26.867 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 21 MB 00:02 2024-05-09T02:59:27.220 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T02:59:27.685 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-base-19.0.0-3603.gc152e9f4.el9.x86 5.3 MB/s | 5.4 MB 00:01 2024-05-09T02:59:27.735 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-selinux-19.0.0-3603.gc152e9f4.el9. 505 kB/s | 25 kB 00:00 2024-05-09T02:59:27.852 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): libcephfs2-19.0.0-3603.gc152e9f4.el9.x8 6.3 MB/s | 747 kB 00:00 2024-05-09T02:59:27.936 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): libradosstriper1-19.0.0-3603.gc152e9f4. 5.8 MB/s | 501 kB 00:00 2024-05-09T02:59:28.503 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): librgw2-19.0.0-3603.gc152e9f4.el9.x86_6 9.5 MB/s | 5.4 MB 00:00 2024-05-09T02:59:28.570 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): python3-ceph-argparse-19.0.0-3603.gc152 679 kB/s | 45 kB 00:00 2024-05-09T02:59:28.629 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-ceph-common-19.0.0-3603.gc152e9 2.3 MB/s | 135 kB 00:00 2024-05-09T02:59:28.737 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): ceph-radosgw-19.0.0-3603.gc152e9f4.el9. 5.1 MB/s | 11 MB 00:02 2024-05-09T02:59:28.762 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-cephfs-19.0.0-3603.gc152e9f4.el 1.2 MB/s | 161 kB 00:00 2024-05-09T02:59:28.813 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-rados-19.0.0-3603.gc152e9f4.el 4.2 MB/s | 320 kB 00:00 2024-05-09T02:59:28.838 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-rbd-19.0.0-3603.gc152e9f4.el9. 3.9 MB/s | 300 kB 00:00 2024-05-09T02:59:28.880 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): python3-rgw-19.0.0-3603.gc152e9f4.el9. 1.5 MB/s | 100 kB 00:00 2024-05-09T02:59:29.177 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T02:59:29.188 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:29.188 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 11 M 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.4 M 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 21 M 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 25 k 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T02:59:29.189 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 747 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 501 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 45 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 135 k 2024-05-09T02:59:29.190 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 161 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 320 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 300 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 100 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout:Install 30 Packages 2024-05-09T02:59:29.191 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-05-09T02:59:29.192 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:29.192 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 60 M 2024-05-09T02:59:29.192 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T02:59:29.347 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): ceph-common-19.0.0-3603.gc152e9f4.el9. 7.9 MB/s | 21 MB 00:02 2024-05-09T02:59:29.665 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 11 M 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.4 M 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T02:59:29.688 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 21 M 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 25 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 747 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 501 k 2024-05-09T02:59:29.689 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 45 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 135 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 161 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 320 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 300 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 100 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T02:59:29.690 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:Install 30 Packages 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 2 Packages 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 60 M 2024-05-09T02:59:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T02:59:29.747 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 118 kB/s | 107 kB 00:00 2024-05-09T02:59:29.806 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 173 kB/s | 160 kB 00:00 2024-05-09T02:59:30.015 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 68 kB/s | 45 kB 00:00 2024-05-09T02:59:30.213 INFO:teuthology.orchestra.run.smithi181.stdout:(1/32): ceph-base-19.0.0-3603.gc152e9f4.el9.x86 6.7 MB/s | 5.4 MB 00:00 2024-05-09T02:59:30.263 INFO:teuthology.orchestra.run.smithi181.stdout:(2/32): ceph-selinux-19.0.0-3603.gc152e9f4.el9. 504 kB/s | 25 kB 00:00 2024-05-09T02:59:30.364 INFO:teuthology.orchestra.run.smithi181.stdout:(3/32): libcephfs2-19.0.0-3603.gc152e9f4.el9.x8 7.3 MB/s | 747 kB 00:00 2024-05-09T02:59:30.472 INFO:teuthology.orchestra.run.smithi181.stdout:(4/32): libradosstriper1-19.0.0-3603.gc152e9f4. 4.5 MB/s | 501 kB 00:00 2024-05-09T02:59:30.589 INFO:teuthology.orchestra.run.smithi181.stdout:(5/32): ceph-radosgw-19.0.0-3603.gc152e9f4.el9. 9.0 MB/s | 11 MB 00:01 2024-05-09T02:59:30.648 INFO:teuthology.orchestra.run.smithi181.stdout:(6/32): python3-ceph-argparse-19.0.0-3603.gc152 778 kB/s | 45 kB 00:00 2024-05-09T02:59:30.715 INFO:teuthology.orchestra.run.smithi181.stdout:(7/32): python3-ceph-common-19.0.0-3603.gc152e9 2.0 MB/s | 135 kB 00:00 2024-05-09T02:59:30.723 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 62 kB/s | 42 kB 00:00 2024-05-09T02:59:30.773 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 302 kB/s | 292 kB 00:00 2024-05-09T02:59:30.799 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 631 kB/s | 662 kB 00:01 2024-05-09T02:59:30.807 INFO:teuthology.orchestra.run.smithi181.stdout:(8/32): python3-cephfs-19.0.0-3603.gc152e9f4.el 1.7 MB/s | 161 kB 00:00 2024-05-09T02:59:30.899 INFO:teuthology.orchestra.run.smithi181.stdout:(9/32): python3-rados-19.0.0-3603.gc152e9f4.el9 3.4 MB/s | 320 kB 00:00 2024-05-09T02:59:30.990 INFO:teuthology.orchestra.run.smithi181.stdout:(10/32): python3-rbd-19.0.0-3603.gc152e9f4.el9. 3.2 MB/s | 300 kB 00:00 2024-05-09T02:59:31.074 INFO:teuthology.orchestra.run.smithi181.stdout:(11/32): librgw2-19.0.0-3603.gc152e9f4.el9.x86_ 8.9 MB/s | 5.4 MB 00:00 2024-05-09T02:59:31.099 INFO:teuthology.orchestra.run.smithi181.stdout:(12/32): python3-rgw-19.0.0-3603.gc152e9f4.el9. 918 kB/s | 100 kB 00:00 2024-05-09T02:59:31.374 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 60 kB/s | 36 kB 00:00 2024-05-09T02:59:31.458 INFO:teuthology.orchestra.run.smithi072.stdout:(1/32): ceph-base-19.0.0-3603.gc152e9f4.el9.x86 4.7 MB/s | 5.4 MB 00:01 2024-05-09T02:59:31.499 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 68 kB/s | 47 kB 00:00 2024-05-09T02:59:31.509 INFO:teuthology.orchestra.run.smithi072.stdout:(2/32): ceph-selinux-19.0.0-3603.gc152e9f4.el9. 501 kB/s | 25 kB 00:00 2024-05-09T02:59:31.591 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 254 kB/s | 221 kB 00:00 2024-05-09T02:59:31.659 INFO:teuthology.orchestra.run.smithi072.stdout:(3/32): libcephfs2-19.0.0-3603.gc152e9f4.el9.x8 4.9 MB/s | 747 kB 00:00 2024-05-09T02:59:31.717 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 903 kB/s | 309 kB 00:00 2024-05-09T02:59:31.775 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-05-09T02:59:31.793 INFO:teuthology.orchestra.run.smithi072.stdout:(4/32): libradosstriper1-19.0.0-3603.gc152e9f4. 3.7 MB/s | 501 kB 00:00 2024-05-09T02:59:31.800 INFO:teuthology.orchestra.run.smithi181.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 147 kB/s | 107 kB 00:00 2024-05-09T02:59:31.800 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 124 kB/s | 26 kB 00:00 2024-05-09T02:59:31.825 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T02:59:31.884 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-09T02:59:31.908 INFO:teuthology.orchestra.run.smithi181.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 198 kB/s | 160 kB 00:00 2024-05-09T02:59:32.048 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-05-09T02:59:32.068 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-05-09T02:59:32.101 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-09T02:59:32.135 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.5 MB/s | 839 kB 00:00 2024-05-09T02:59:32.210 INFO:teuthology.orchestra.run.smithi072.stdout:(5/32): ceph-radosgw-19.0.0-3603.gc152e9f4.el9. 5.6 MB/s | 11 MB 00:01 2024-05-09T02:59:32.269 INFO:teuthology.orchestra.run.smithi072.stdout:(6/32): python3-ceph-argparse-19.0.0-3603.gc152 773 kB/s | 45 kB 00:00 2024-05-09T02:59:32.345 INFO:teuthology.orchestra.run.smithi072.stdout:(7/32): python3-ceph-common-19.0.0-3603.gc152e9 1.8 MB/s | 135 kB 00:00 2024-05-09T02:59:32.350 INFO:teuthology.orchestra.run.smithi181.stdout:(15/32): ceph-common-19.0.0-3603.gc152e9f4.el9. 7.2 MB/s | 21 MB 00:02 2024-05-09T02:59:32.470 INFO:teuthology.orchestra.run.smithi072.stdout:(8/32): librgw2-19.0.0-3603.gc152e9f4.el9.x86_6 7.9 MB/s | 5.4 MB 00:00 2024-05-09T02:59:32.505 INFO:teuthology.orchestra.run.smithi072.stdout:(9/32): python3-cephfs-19.0.0-3603.gc152e9f4.el 1.0 MB/s | 161 kB 00:00 2024-05-09T02:59:32.538 INFO:teuthology.orchestra.run.smithi072.stdout:(10/32): python3-rados-19.0.0-3603.gc152e9f4.el 4.6 MB/s | 320 kB 00:00 2024-05-09T02:59:32.572 INFO:teuthology.orchestra.run.smithi072.stdout:(11/32): python3-rbd-19.0.0-3603.gc152e9f4.el9. 4.3 MB/s | 300 kB 00:00 2024-05-09T02:59:32.576 INFO:teuthology.orchestra.run.smithi181.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 58 kB/s | 45 kB 00:00 2024-05-09T02:59:32.585 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): librados2-19.0.0-3603.gc152e9f4.el9.x8 6.6 MB/s | 3.4 MB 00:00 2024-05-09T02:59:32.599 INFO:teuthology.orchestra.run.smithi072.stdout:(12/32): python3-rgw-19.0.0-3603.gc152e9f4.el9. 1.6 MB/s | 100 kB 00:00 2024-05-09T02:59:32.716 INFO:teuthology.orchestra.run.smithi072.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 742 kB/s | 107 kB 00:00 2024-05-09T02:59:32.742 INFO:teuthology.orchestra.run.smithi072.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-05-09T02:59:32.768 INFO:teuthology.orchestra.run.smithi072.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-05-09T02:59:32.810 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): librbd1-19.0.0-3603.gc152e9f4.el9.x86_ 4.4 MB/s | 3.1 MB 00:00 2024-05-09T02:59:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:59:32.816 INFO:teuthology.orchestra.run.smithi195.stdout:Total 9.2 MB/s | 60 MB 00:06 2024-05-09T02:59:32.843 INFO:teuthology.orchestra.run.smithi072.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-05-09T02:59:32.869 INFO:teuthology.orchestra.run.smithi072.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-05-09T02:59:32.894 INFO:teuthology.orchestra.run.smithi072.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 42 kB 00:00 2024-05-09T02:59:32.925 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T02:59:32.945 INFO:teuthology.orchestra.run.smithi072.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 712 kB/s | 36 kB 00:00 2024-05-09T02:59:32.951 INFO:teuthology.orchestra.run.smithi181.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 636 kB/s | 662 kB 00:01 2024-05-09T02:59:32.962 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T02:59:32.962 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T02:59:32.995 INFO:teuthology.orchestra.run.smithi072.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-09T02:59:33.030 INFO:teuthology.orchestra.run.smithi072.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.3 MB/s | 221 kB 00:00 2024-05-09T02:59:33.197 INFO:teuthology.orchestra.run.smithi072.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-05-09T02:59:33.239 INFO:teuthology.orchestra.run.smithi072.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 618 kB/s | 26 kB 00:00 2024-05-09T02:59:33.243 INFO:teuthology.orchestra.run.smithi181.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 328 kB/s | 292 kB 00:00 2024-05-09T02:59:33.268 INFO:teuthology.orchestra.run.smithi181.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 63 kB/s | 42 kB 00:00 2024-05-09T02:59:33.281 INFO:teuthology.orchestra.run.smithi072.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-05-09T02:59:33.331 INFO:teuthology.orchestra.run.smithi072.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T02:59:33.501 INFO:teuthology.orchestra.run.smithi072.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-05-09T02:59:33.576 INFO:teuthology.orchestra.run.smithi072.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-09T02:59:33.593 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T02:59:33.593 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T02:59:33.768 INFO:teuthology.orchestra.run.smithi181.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 271 kB/s | 221 kB 00:00 2024-05-09T02:59:33.868 INFO:teuthology.orchestra.run.smithi181.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 57 kB/s | 36 kB 00:00 2024-05-09T02:59:33.919 INFO:teuthology.orchestra.run.smithi181.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-05-09T02:59:33.961 INFO:teuthology.orchestra.run.smithi181.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-05-09T02:59:33.977 INFO:teuthology.orchestra.run.smithi072.stdout:(28/32): ceph-common-19.0.0-3603.gc152e9f4.el9. 5.8 MB/s | 21 MB 00:03 2024-05-09T02:59:34.136 INFO:teuthology.orchestra.run.smithi072.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-05-09T02:59:34.228 INFO:teuthology.orchestra.run.smithi072.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-05-09T02:59:34.387 INFO:teuthology.orchestra.run.smithi072.stdout:(31/32): librados2-19.0.0-3603.gc152e9f4.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-05-09T02:59:34.439 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T02:59:34.442 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:59:34.442 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 11 M 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.4 M 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 21 M 2024-05-09T02:59:34.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 25 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 747 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 501 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 5.4 M 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T02:59:34.444 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 45 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 135 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 161 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 320 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 300 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 100 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T02:59:34.445 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout:Install 30 Packages 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:59:34.446 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 60 M 2024-05-09T02:59:34.447 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T02:59:34.504 INFO:teuthology.orchestra.run.smithi072.stdout:(32/32): librbd1-19.0.0-3603.gc152e9f4.el9.x86_ 8.5 MB/s | 3.1 MB 00:00 2024-05-09T02:59:34.506 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:59:34.506 INFO:teuthology.orchestra.run.smithi072.stdout:Total 13 MB/s | 60 MB 00:04 2024-05-09T02:59:34.511 INFO:teuthology.orchestra.run.smithi181.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 44 kB/s | 26 kB 00:00 2024-05-09T02:59:34.536 INFO:teuthology.orchestra.run.smithi181.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 84 kB/s | 49 kB 00:00 2024-05-09T02:59:34.615 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T02:59:34.650 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T02:59:34.651 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T02:59:34.720 INFO:teuthology.orchestra.run.smithi181.stdout:(26/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-05-09T02:59:34.762 INFO:teuthology.orchestra.run.smithi181.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 300 kB/s | 67 kB 00:00 2024-05-09T02:59:34.803 INFO:teuthology.orchestra.run.smithi181.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 839 kB 00:00 2024-05-09T02:59:34.845 INFO:teuthology.orchestra.run.smithi181.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-05-09T02:59:34.879 INFO:teuthology.orchestra.run.smithi181.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-09T02:59:35.013 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T02:59:35.103 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-09T02:59:35.133 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-09T02:59:35.157 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-09T02:59:35.180 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:35.211 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:35.278 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T02:59:35.279 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T02:59:35.279 INFO:teuthology.orchestra.run.smithi181.stdout:(31/32): librados2-19.0.0-3603.gc152e9f4.el9.x8 7.1 MB/s | 3.4 MB 00:00 2024-05-09T02:59:35.399 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:35.863 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-09T02:59:35.917 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:36.010 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 8/34 2024-05-09T02:59:36.136 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:36.198 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:36.227 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-09T02:59:36.238 INFO:teuthology.orchestra.run.smithi181.stdout:(32/32): librbd1-19.0.0-3603.gc152e9f4.el9.x86_ 2.3 MB/s | 3.1 MB 00:01 2024-05-09T02:59:36.243 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:59:36.243 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.6 MB/s | 60 MB 00:07 2024-05-09T02:59:36.249 INFO:teuthology.orchestra.run.smithi096.stdout:(1/32): ceph-common-19.0.0-3603.gc152e9f4.el9.x 13 MB/s | 21 MB 00:01 2024-05-09T02:59:36.271 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 11/34 2024-05-09T02:59:36.299 INFO:teuthology.orchestra.run.smithi096.stdout:(2/32): ceph-selinux-19.0.0-3603.gc152e9f4.el9. 505 kB/s | 25 kB 00:00 2024-05-09T02:59:36.340 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:36.350 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T02:59:36.363 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:36.374 INFO:teuthology.orchestra.run.smithi096.stdout:(3/32): libcephfs2-19.0.0-3603.gc152e9f4.el9.x8 9.8 MB/s | 747 kB 00:00 2024-05-09T02:59:36.387 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T02:59:36.388 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T02:59:36.419 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:36.441 INFO:teuthology.orchestra.run.smithi096.stdout:(4/32): libradosstriper1-19.0.0-3603.gc152e9f4. 7.3 MB/s | 501 kB 00:00 2024-05-09T02:59:36.453 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-09T02:59:36.569 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-09T02:59:36.595 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-09T02:59:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-09T02:59:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-09T02:59:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-09T02:59:36.700 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T02:59:36.731 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-09T02:59:36.775 INFO:teuthology.orchestra.run.smithi096.stdout:(5/32): librgw2-19.0.0-3603.gc152e9f4.el9.x86_6 16 MB/s | 5.4 MB 00:00 2024-05-09T02:59:36.790 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-09T02:59:36.816 INFO:teuthology.orchestra.run.smithi096.stdout:(6/32): python3-ceph-argparse-19.0.0-3603.gc152 1.1 MB/s | 45 kB 00:00 2024-05-09T02:59:36.828 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-09T02:59:36.867 INFO:teuthology.orchestra.run.smithi096.stdout:(7/32): python3-ceph-common-19.0.0-3603.gc152e9 2.7 MB/s | 135 kB 00:00 2024-05-09T02:59:36.872 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-09T02:59:36.903 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:36.926 INFO:teuthology.orchestra.run.smithi096.stdout:(8/32): python3-cephfs-19.0.0-3603.gc152e9f4.el 2.7 MB/s | 161 kB 00:00 2024-05-09T02:59:36.931 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:36.993 INFO:teuthology.orchestra.run.smithi096.stdout:(9/32): python3-rados-19.0.0-3603.gc152e9f4.el9 4.7 MB/s | 320 kB 00:00 2024-05-09T02:59:37.007 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T02:59:37.008 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T02:59:37.043 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-09T02:59:37.051 INFO:teuthology.orchestra.run.smithi096.stdout:(10/32): python3-rbd-19.0.0-3603.gc152e9f4.el9. 5.0 MB/s | 300 kB 00:00 2024-05-09T02:59:37.088 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:37.122 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:37.160 INFO:teuthology.orchestra.run.smithi096.stdout:(11/32): ceph-radosgw-19.0.0-3603.gc152e9f4.el9 4.2 MB/s | 11 MB 00:02 2024-05-09T02:59:37.182 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:37.185 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:37.185 INFO:teuthology.orchestra.run.smithi096.stdout:(12/32): python3-rgw-19.0.0-3603.gc152e9f4.el9. 746 kB/s | 100 kB 00:00 2024-05-09T02:59:37.216 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 23/34 2024-05-09T02:59:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-09T02:59:37.378 INFO:teuthology.orchestra.run.smithi096.stdout:(13/32): ceph-base-19.0.0-3603.gc152e9f4.el9.x8 2.0 MB/s | 5.4 MB 00:02 2024-05-09T02:59:37.538 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-09T02:59:37.729 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:37.731 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-09T02:59:37.755 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:37.809 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:37.864 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:37.878 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 27/34 2024-05-09T02:59:37.915 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 28/34 2024-05-09T02:59:37.944 INFO:teuthology.orchestra.run.smithi096.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 136 kB/s | 107 kB 00:00 2024-05-09T02:59:37.953 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 8/34 2024-05-09T02:59:37.983 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:37.986 INFO:teuthology.orchestra.run.smithi096.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 74 kB/s | 45 kB 00:00 2024-05-09T02:59:38.028 INFO:teuthology.orchestra.run.smithi096.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 190 kB/s | 160 kB 00:00 2024-05-09T02:59:38.046 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:38.075 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-09T02:59:38.119 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 11/34 2024-05-09T02:59:38.199 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:38.230 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:38.330 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-09T02:59:38.407 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T02:59:38.456 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-09T02:59:38.490 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-09T02:59:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-09T02:59:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-09T02:59:38.534 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-09T02:59:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-09T02:59:38.565 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-09T02:59:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:38.604 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-09T02:59:38.637 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-09T02:59:38.678 INFO:teuthology.orchestra.run.smithi096.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 65 kB/s | 42 kB 00:00 2024-05-09T02:59:38.793 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:38.836 INFO:teuthology.orchestra.run.smithi096.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 344 kB/s | 292 kB 00:00 2024-05-09T02:59:38.857 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:38.950 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-09T02:59:39.002 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:39.045 INFO:teuthology.orchestra.run.smithi096.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 602 kB/s | 662 kB 00:01 2024-05-09T02:59:39.092 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:39.124 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 23/34 2024-05-09T02:59:39.169 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-09T02:59:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-09T02:59:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:39.421 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:39.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 8/34 2024-05-09T02:59:39.547 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:39.603 INFO:teuthology.orchestra.run.smithi096.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 47 kB/s | 36 kB 00:00 2024-05-09T02:59:39.609 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:39.638 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-09T02:59:39.651 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-09T02:59:39.682 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 11/34 2024-05-09T02:59:39.683 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:39.690 INFO:teuthology.orchestra.run.smithi096.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 219 kB/s | 221 kB 00:01 2024-05-09T02:59:39.713 INFO:teuthology.orchestra.run.smithi096.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-05-09T02:59:39.738 INFO:teuthology.orchestra.run.smithi096.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 MB/s | 309 kB 00:00 2024-05-09T02:59:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:39.754 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:39.771 INFO:teuthology.orchestra.run.smithi096.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-05-09T02:59:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:39.797 INFO:teuthology.orchestra.run.smithi096.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 310 kB/s | 26 kB 00:00 2024-05-09T02:59:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:39.822 INFO:teuthology.orchestra.run.smithi096.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T02:59:39.832 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 27/34 2024-05-09T02:59:39.855 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-09T02:59:39.856 INFO:teuthology.orchestra.run.smithi096.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-05-09T02:59:39.871 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 28/34 2024-05-09T02:59:39.939 INFO:teuthology.orchestra.run.smithi096.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-09T02:59:40.006 INFO:teuthology.orchestra.run.smithi096.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-05-09T02:59:40.039 INFO:teuthology.orchestra.run.smithi096.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-05-09T02:59:40.083 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-09T02:59:40.115 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-09T02:59:40.155 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-09T02:59:40.188 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-09T02:59:40.217 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-09T02:59:40.236 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:40.249 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-09T02:59:40.273 INFO:teuthology.orchestra.run.smithi096.stdout:(31/32): librbd1-19.0.0-3603.gc152e9f4.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-05-09T02:59:40.279 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:40.562 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-09T02:59:40.613 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:40.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 23/34 2024-05-09T02:59:40.779 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-09T02:59:40.851 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:40.891 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:40.940 INFO:teuthology.orchestra.run.smithi096.stdout:(32/32): librados2-19.0.0-3603.gc152e9f4.el9.x8 3.4 MB/s | 3.4 MB 00:01 2024-05-09T02:59:40.943 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:59:40.943 INFO:teuthology.orchestra.run.smithi096.stdout:Total 9.3 MB/s | 60 MB 00:06 2024-05-09T02:59:40.983 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:40.983 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T02:59:40.983 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T02:59:40.983 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:41.018 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:41.048 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T02:59:41.084 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T02:59:41.084 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T02:59:41.258 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-09T02:59:41.282 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:41.353 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:41.422 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 27/34 2024-05-09T02:59:41.451 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 28/34 2024-05-09T02:59:41.719 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T02:59:41.719 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T02:59:42.177 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:42.224 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:42.790 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:42.833 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:42.933 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:42.934 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T02:59:42.934 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T02:59:42.934 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:42.971 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T02:59:43.224 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-09T02:59:43.254 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-09T02:59:43.287 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-09T02:59:43.310 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:43.336 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-09T02:59:43.528 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:43.582 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-09T02:59:43.754 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:43.910 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-09T02:59:44.005 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:44.097 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:44.187 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 8/34 2024-05-09T02:59:44.215 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:44.280 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 9/34 2024-05-09T02:59:44.315 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-09T02:59:44.371 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:44.378 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 11/34 2024-05-09T02:59:44.409 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:44.453 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:44.476 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:44.499 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:44.499 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T02:59:44.499 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T02:59:44.499 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:44.535 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:44.540 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 13/34 2024-05-09T02:59:44.574 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-09T02:59:44.693 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-09T02:59:44.734 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-09T02:59:44.775 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-09T02:59:44.807 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-09T02:59:44.836 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-09T02:59:44.860 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-09T02:59:45.183 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-09T02:59:45.223 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:45.298 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 22/34 2024-05-09T02:59:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 23/34 2024-05-09T02:59:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-09T02:59:45.845 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-09T02:59:45.876 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:45.947 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 26/34 2024-05-09T02:59:46.025 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 27/34 2024-05-09T02:59:46.053 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 28/34 2024-05-09T02:59:48.358 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:48.416 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:49.010 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 29/34 2024-05-09T02:59:49.053 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:49.155 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 30/34 2024-05-09T02:59:49.155 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T02:59:49.155 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T02:59:49.155 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:59:49.188 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-05-09T02:59:50.586 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:51.579 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:51.604 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:51.606 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:51.690 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:51.690 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:52.799 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 6/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 8/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 9/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 10/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 11/34 2024-05-09T02:59:52.800 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 13/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-09T02:59:52.801 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-09T02:59:52.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/34 2024-05-09T02:59:53.649 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.650 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.651 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:53.652 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T02:59:53.844 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-05-09T02:59:53.845 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:54.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-05-09T02:59:54.593 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:54.598 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:31 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T02:59:54.718 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 49 M 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 163 k 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T02:59:54.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 50 M 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 211 M 2024-05-09T02:59:54.720 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T02:59:54.854 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:54.879 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:54.879 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T02:59:54.880 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T02:59:54.880 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:54.880 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:54.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:54.881 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:54.948 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:54.949 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:54.961 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-05-09T02:59:54.994 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-05-09T02:59:55.170 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): libcephsqlite-19.0.0-3603.gc152e9f4.el9. 437 kB/s | 163 kB 00:00 2024-05-09T02:59:55.618 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:55.643 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:55.643 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T02:59:55.643 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T02:59:55.643 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:55.644 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:55.644 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:55.645 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:55.756 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:55.756 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:56.057 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 6/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 8/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 9/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 10/34 2024-05-09T02:59:56.058 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 11/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 13/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-09T02:59:56.059 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-09T02:59:56.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/34 2024-05-09T02:59:56.990 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:56.993 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T02:59:57.154 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): ceph-test-19.0.0-3603.gc152e9f4.el9.x86_ 21 MB/s | 49 MB 00:02 2024-05-09T02:59:57.156 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:59:57.157 INFO:teuthology.orchestra.run.smithi195.stdout:Total 20 MB/s | 50 MB 00:02 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 6/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 8/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 9/34 2024-05-09T02:59:57.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 10/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 11/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 13/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-09T02:59:57.161 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-09T02:59:57.162 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/34 2024-05-09T02:59:57.184 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T02:59:57.204 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T02:59:57.204 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T02:59:57.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-05-09T02:59:57.576 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T02:59:57.577 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T02:59:57.916 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:31 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T02:59:57.938 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T02:59:57.938 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:57.938 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-05-09T02:59:57.938 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T02:59:57.939 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T02:59:57.940 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:57.941 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T02:59:58.037 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T02:59:58.039 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:58.039 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T02:59:58.039 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 49 M 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 163 k 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 50 M 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 211 M 2024-05-09T02:59:58.040 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T02:59:58.107 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T02:59:58.188 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-09T02:59:58.228 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-09T02:59:58.255 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T02:59:58.322 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-test 2024-05-09T02:59:58.545 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): libcephsqlite-19.0.0-3603.gc152e9f4.el9. 450 kB/s | 163 kB 00:00 2024-05-09T02:59:58.829 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T02:59:58.829 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-05-09T02:59:58.829 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-05-09T02:59:58.829 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-05-09T02:59:58.829 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-05-09T02:59:58.830 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-05-09T02:59:58.830 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-05-09T02:59:58.830 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-05-09T02:59:58.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:59:58.891 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:31 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T02:59:59.013 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T02:59:59.013 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:59.013 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T02:59:59.013 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 49 M 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 163 k 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T02:59:59.014 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-05-09T02:59:59.015 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T02:59:59.015 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 50 M 2024-05-09T02:59:59.015 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 211 M 2024-05-09T02:59:59.015 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T02:59:59.020 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 364 kB/s | 305 kB 00:00 2024-05-09T02:59:59.204 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-05-09T02:59:59.620 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 979 kB/s | 305 kB 00:00 2024-05-09T02:59:59.687 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 952 kB/s | 64 kB 00:00 2024-05-09T02:59:59.805 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:59.831 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/34 2024-05-09T02:59:59.831 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T02:59:59.831 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T02:59:59.831 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:59.832 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T02:59:59.832 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T02:59:59.833 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:59.871 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): libcephsqlite-19.0.0-3603.gc152e9f4.el9. 291 kB/s | 163 kB 00:00 2024-05-09T02:59:59.916 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-09T02:59:59.916 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T03:00:00.737 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-19.0.0-3603.gc152e9f4.el9.x86_ 19 MB/s | 49 MB 00:02 2024-05-09T03:00:00.739 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:00.740 INFO:teuthology.orchestra.run.smithi181.stdout:Total 18 MB/s | 50 MB 00:02 2024-05-09T03:00:00.768 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:00.788 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:00.788 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:01.085 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T03:00:01.085 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_6 6/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 7/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9. 8/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x8 9/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 10/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 11/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 12/34 2024-05-09T03:00:01.086 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 13/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-09T03:00:01.087 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 31/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-09T03:00:01.088 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/34 2024-05-09T03:00:01.160 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:01.160 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:01.681 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:01.757 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-09T03:00:01.803 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-09T03:00:01.831 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:01.884 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-09T03:00:01.884 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:01.884 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-05-09T03:00:01.884 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.884 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.885 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T03:00:01.886 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T03:00:01.887 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T03:00:01.887 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T03:00:01.887 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T03:00:01.887 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:01.887 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:02.261 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-05-09T03:00:02.833 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:30 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:02.907 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 49 M 2024-05-09T03:00:02.955 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 163 k 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-05-09T03:00:02.956 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:02.957 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 50 M 2024-05-09T03:00:02.957 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 211 M 2024-05-09T03:00:02.957 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:02.986 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:03.138 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-05-09T03:00:03.147 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): ceph-test-19.0.0-3603.gc152e9f4.el9.x86_ 13 MB/s | 49 MB 00:03 2024-05-09T03:00:03.149 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:03.149 INFO:teuthology.orchestra.run.smithi072.stdout:Total 12 MB/s | 50 MB 00:04 2024-05-09T03:00:03.177 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:03.196 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-09T03:00:03.197 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:03.197 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:03.405 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): libcephsqlite-19.0.0-3603.gc152e9f4.el9. 405 kB/s | 163 kB 00:00 2024-05-09T03:00:03.568 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:03.569 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:04.088 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:04.168 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-09T03:00:04.224 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-09T03:00:04.252 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:05.364 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:05.364 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/4 2024-05-09T03:00:05.364 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/4 2024-05-09T03:00:05.364 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-09T03:00:05.538 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): ceph-test-19.0.0-3603.gc152e9f4.el9.x86_ 19 MB/s | 49 MB 00:02 2024-05-09T03:00:05.540 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:05.541 INFO:teuthology.orchestra.run.smithi096.stdout:Total 19 MB/s | 50 MB 00:02 2024-05-09T03:00:05.569 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:05.589 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:05.589 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:05.627 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-09T03:00:05.627 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:05.627 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:05.628 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:05.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-05-09T03:00:05.961 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:05.962 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:06.460 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:43 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:06.480 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:06.568 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-09T03:00:06.569 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:06.591 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:06.593 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 6.6 k 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 2.4 M 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 1.5 M 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 274 k 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 4.7 M 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 17 M 2024-05-09T03:00:06.594 INFO:teuthology.orchestra.run.smithi195.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:00:06.595 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:00:06.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout:Install 36 Packages 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 30 M 2024-05-09T03:00:06.597 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 108 M 2024-05-09T03:00:06.598 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:06.613 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-09T03:00:06.640 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:06.645 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:07.589 INFO:teuthology.orchestra.run.smithi195.stdout:(1/36): ceph-19.0.0-3603.gc152e9f4.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-05-09T03:00:08.031 INFO:teuthology.orchestra.run.smithi195.stdout:(2/36): ceph-mon-19.0.0-3603.gc152e9f4.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-05-09T03:00:08.106 INFO:teuthology.orchestra.run.smithi195.stdout:(3/36): ceph-mgr-19.0.0-3603.gc152e9f4.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-05-09T03:00:08.215 INFO:teuthology.orchestra.run.smithi195.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3603.gc152 2.5 MB/s | 274 kB 00:00 2024-05-09T03:00:08.323 INFO:teuthology.orchestra.run.smithi195.stdout:(5/36): ceph-mds-19.0.0-3603.gc152e9f4.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-05-09T03:00:08.749 INFO:teuthology.orchestra.run.smithi195.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 475 kB/s | 253 kB 00:00 2024-05-09T03:00:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:08.932 INFO:teuthology.orchestra.run.smithi195.stdout:(7/36): ceph-osd-19.0.0-3603.gc152e9f4.el9.x86_ 19 MB/s | 17 MB 00:00 2024-05-09T03:00:08.958 INFO:teuthology.orchestra.run.smithi195.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 510 kB/s | 106 kB 00:00 2024-05-09T03:00:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/4 2024-05-09T03:00:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/4 2024-05-09T03:00:09.025 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-09T03:00:09.066 INFO:teuthology.orchestra.run.smithi195.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-05-09T03:00:09.175 INFO:teuthology.orchestra.run.smithi195.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-05-09T03:00:09.209 INFO:teuthology.orchestra.run.smithi195.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2024-05-09T03:00:09.259 INFO:teuthology.orchestra.run.smithi195.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-05-09T03:00:09.284 INFO:teuthology.orchestra.run.smithi195.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-05-09T03:00:09.301 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-09T03:00:09.301 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:09.301 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:09.301 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:09.301 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:09.302 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:00:09.302 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:00:09.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:09.302 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:09.318 INFO:teuthology.orchestra.run.smithi195.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 592 kB/s | 35 kB 00:00 2024-05-09T03:00:09.343 INFO:teuthology.orchestra.run.smithi195.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 868 kB/s | 51 kB 00:00 2024-05-09T03:00:09.368 INFO:teuthology.orchestra.run.smithi195.stdout:(16/36): python3-pycparser-2.20-6.el9.noarch.rp 310 kB/s | 135 kB 00:00 2024-05-09T03:00:09.385 INFO:teuthology.orchestra.run.smithi195.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 620 kB/s | 42 kB 00:00 2024-05-09T03:00:09.461 INFO:teuthology.orchestra.run.smithi195.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-05-09T03:00:09.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-05-09T03:00:09.836 INFO:teuthology.orchestra.run.smithi195.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 96 kB/s | 43 kB 00:00 2024-05-09T03:00:10.119 INFO:teuthology.orchestra.run.smithi195.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 261 kB/s | 172 kB 00:00 2024-05-09T03:00:10.140 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:44 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:10.144 INFO:teuthology.orchestra.run.smithi195.stdout:(21/36): luarocks-3.9.2-1.el9.noarch.rpm 195 kB/s | 151 kB 00:00 2024-05-09T03:00:10.236 INFO:teuthology.orchestra.run.smithi195.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-05-09T03:00:10.253 INFO:teuthology.orchestra.run.smithi195.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-05-09T03:00:10.270 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:10.273 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:10.273 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 6.6 k 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 2.4 M 2024-05-09T03:00:10.274 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 1.5 M 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 274 k 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 4.7 M 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 17 M 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:00:10.275 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:00:10.276 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:00:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:00:10.278 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:00:10.279 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:10.280 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:10.281 INFO:teuthology.orchestra.run.smithi181.stdout:Install 36 Packages 2024-05-09T03:00:10.281 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:10.282 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 30 M 2024-05-09T03:00:10.282 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 108 M 2024-05-09T03:00:10.282 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:10.354 INFO:teuthology.orchestra.run.smithi195.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-05-09T03:00:10.398 INFO:teuthology.orchestra.run.smithi195.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 134 kB/s | 19 kB 00:00 2024-05-09T03:00:10.463 INFO:teuthology.orchestra.run.smithi195.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-05-09T03:00:10.513 INFO:teuthology.orchestra.run.smithi195.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 404 kB/s | 46 kB 00:00 2024-05-09T03:00:10.613 INFO:teuthology.orchestra.run.smithi195.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 461 kB/s | 358 kB 00:00 2024-05-09T03:00:10.663 INFO:teuthology.orchestra.run.smithi195.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 393 kB/s | 79 kB 00:00 2024-05-09T03:00:10.730 INFO:teuthology.orchestra.run.smithi195.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-05-09T03:00:10.847 INFO:teuthology.orchestra.run.smithi195.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 307 kB/s | 36 kB 00:00 2024-05-09T03:00:10.873 INFO:teuthology.orchestra.run.smithi195.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 431 kB/s | 90 kB 00:00 2024-05-09T03:00:11.057 INFO:teuthology.orchestra.run.smithi195.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 501 kB/s | 272 kB 00:00 2024-05-09T03:00:11.173 INFO:teuthology.orchestra.run.smithi195.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-05-09T03:00:11.181 INFO:teuthology.orchestra.run.smithi181.stdout:(1/36): ceph-19.0.0-3603.gc152e9f4.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-05-09T03:00:11.232 INFO:teuthology.orchestra.run.smithi195.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 598 kB/s | 230 kB 00:00 2024-05-09T03:00:11.298 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/4 2024-05-09T03:00:11.331 INFO:teuthology.orchestra.run.smithi181.stdout:(2/36): ceph-mds-19.0.0-3603.gc152e9f4.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-05-09T03:00:11.401 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:11.481 INFO:teuthology.orchestra.run.smithi181.stdout:(3/36): ceph-mgr-19.0.0-3603.gc152e9f4.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-05-09T03:00:11.590 INFO:teuthology.orchestra.run.smithi181.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3603.gc152 2.5 MB/s | 274 kB 00:00 2024-05-09T03:00:11.598 INFO:teuthology.orchestra.run.smithi195.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 589 kB/s | 427 kB 00:00 2024-05-09T03:00:11.600 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:11.600 INFO:teuthology.orchestra.run.smithi195.stdout:Total 6.0 MB/s | 30 MB 00:05 2024-05-09T03:00:11.740 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:11.780 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:11.780 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:12.140 INFO:teuthology.orchestra.run.smithi181.stdout:(5/36): ceph-osd-19.0.0-3603.gc152e9f4.el9.x86_ 21 MB/s | 17 MB 00:00 2024-05-09T03:00:12.222 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:12.223 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:12.224 INFO:teuthology.orchestra.run.smithi181.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 400 kB/s | 253 kB 00:00 2024-05-09T03:00:12.340 INFO:teuthology.orchestra.run.smithi181.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 914 kB/s | 106 kB 00:00 2024-05-09T03:00:12.457 INFO:teuthology.orchestra.run.smithi181.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-05-09T03:00:12.566 INFO:teuthology.orchestra.run.smithi181.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-05-09T03:00:12.749 INFO:teuthology.orchestra.run.smithi181.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-05-09T03:00:12.860 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:12.860 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/4 2024-05-09T03:00:12.860 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/4 2024-05-09T03:00:12.860 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-09T03:00:12.925 INFO:teuthology.orchestra.run.smithi181.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-05-09T03:00:12.983 INFO:teuthology.orchestra.run.smithi181.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 1.5 MB/s | 1.2 MB 00:00 2024-05-09T03:00:13.042 INFO:teuthology.orchestra.run.smithi181.stdout:(13/36): ceph-mon-19.0.0-3603.gc152e9f4.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-05-09T03:00:13.067 INFO:teuthology.orchestra.run.smithi181.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-05-09T03:00:13.117 INFO:teuthology.orchestra.run.smithi181.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-05-09T03:00:13.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-09T03:00:13.127 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:13.127 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:13.128 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:13.143 INFO:teuthology.orchestra.run.smithi181.stdout:(16/36): python3-markupsafe-1.1.1-12.el9.x86_64 218 kB/s | 35 kB 00:00 2024-05-09T03:00:13.168 INFO:teuthology.orchestra.run.smithi181.stdout:(17/36): python3-pytz-2021.1-5.el9.noarch.rpm 406 kB/s | 51 kB 00:00 2024-05-09T03:00:13.293 INFO:teuthology.orchestra.run.smithi181.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-05-09T03:00:13.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph 2024-05-09T03:00:13.452 INFO:teuthology.orchestra.run.smithi181.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-05-09T03:00:13.507 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:13.560 INFO:teuthology.orchestra.run.smithi181.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 362 kB/s | 151 kB 00:00 2024-05-09T03:00:13.568 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-09T03:00:13.601 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-09T03:00:13.629 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-09T03:00:13.636 INFO:teuthology.orchestra.run.smithi181.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-05-09T03:00:13.661 INFO:teuthology.orchestra.run.smithi181.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 469 kB/s | 172 kB 00:00 2024-05-09T03:00:13.695 INFO:teuthology.orchestra.run.smithi181.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-05-09T03:00:13.701 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-09T03:00:13.720 INFO:teuthology.orchestra.run.smithi181.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-05-09T03:00:13.744 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-09T03:00:13.745 INFO:teuthology.orchestra.run.smithi181.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 275 kB/s | 23 kB 00:00 2024-05-09T03:00:13.759 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/4 2024-05-09T03:00:13.759 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/4 2024-05-09T03:00:13.759 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/4 2024-05-09T03:00:13.759 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-09T03:00:13.770 INFO:teuthology.orchestra.run.smithi181.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-05-09T03:00:13.777 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-09T03:00:13.796 INFO:teuthology.orchestra.run.smithi181.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-05-09T03:00:13.821 INFO:teuthology.orchestra.run.smithi181.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-05-09T03:00:13.846 INFO:teuthology.orchestra.run.smithi181.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-05-09T03:00:13.868 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-09T03:00:13.888 INFO:teuthology.orchestra.run.smithi181.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-05-09T03:00:13.922 INFO:teuthology.orchestra.run.smithi181.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-05-09T03:00:13.947 INFO:teuthology.orchestra.run.smithi181.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-05-09T03:00:13.948 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-09T03:00:13.972 INFO:teuthology.orchestra.run.smithi181.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-05-09T03:00:13.988 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-09T03:00:13.998 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:46 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:14.014 INFO:teuthology.orchestra.run.smithi181.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-09T03:00:14.023 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:00:14.028 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:00:14.029 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:14.029 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-09T03:00:14.063 INFO:teuthology.orchestra.run.smithi181.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 221 kB/s | 20 kB 00:00 2024-05-09T03:00:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-09T03:00:14.090 INFO:teuthology.orchestra.run.smithi181.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-05-09T03:00:14.094 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:14.097 INFO:teuthology.orchestra.run.smithi181.stdout:Total 7.9 MB/s | 30 MB 00:03 2024-05-09T03:00:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-09T03:00:14.122 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 6.6 k 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:00:14.125 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 2.4 M 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 1.5 M 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 274 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 4.7 M 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 17 M 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:00:14.126 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:00:14.127 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout:Install 36 Packages 2024-05-09T03:00:14.128 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:14.129 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 30 M 2024-05-09T03:00:14.129 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 108 M 2024-05-09T03:00:14.129 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:00:14.153 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-09T03:00:14.197 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-09T03:00:14.231 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-09T03:00:14.239 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:14.269 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-09T03:00:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:14.279 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:14.320 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-05-09T03:00:14.343 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-09T03:00:14.489 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-09T03:00:14.569 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-09T03:00:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-09T03:00:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-09T03:00:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-09T03:00:14.718 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:14.718 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:14.760 INFO:teuthology.orchestra.run.smithi072.stdout:(1/36): ceph-19.0.0-3603.gc152e9f4.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-05-09T03:00:14.774 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-09T03:00:14.896 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:42 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:14.979 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-09T03:00:15.023 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-09T03:00:15.026 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 6.6 k 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 2.4 M 2024-05-09T03:00:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 1.5 M 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 274 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 4.7 M 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 17 M 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:00:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:00:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout:Install 36 Packages 2024-05-09T03:00:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:15.033 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 30 M 2024-05-09T03:00:15.033 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 108 M 2024-05-09T03:00:15.033 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:15.094 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-09T03:00:15.191 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-09T03:00:15.238 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-09T03:00:15.364 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-09T03:00:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 31/36 2024-05-09T03:00:15.536 INFO:teuthology.orchestra.run.smithi072.stdout:(2/36): ceph-mds-19.0.0-3603.gc152e9f4.el9.x86_ 2.3 MB/s | 2.4 MB 00:01 2024-05-09T03:00:15.540 INFO:teuthology.orchestra.run.smithi096.stdout:(1/36): ceph-19.0.0-3603.gc152e9f4.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-05-09T03:00:15.557 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:15.584 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:15.586 INFO:teuthology.orchestra.run.smithi072.stdout:(3/36): ceph-mgr-19.0.0-3603.gc152e9f4.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-05-09T03:00:15.679 INFO:teuthology.orchestra.run.smithi072.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3603.gc152 2.9 MB/s | 274 kB 00:00 2024-05-09T03:00:15.846 INFO:teuthology.orchestra.run.smithi072.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-05-09T03:00:15.968 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:15.973 INFO:teuthology.orchestra.run.smithi096.stdout:(2/36): ceph-mgr-19.0.0-3603.gc152e9f4.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-05-09T03:00:16.026 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-09T03:00:16.067 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-09T03:00:16.095 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-09T03:00:16.113 INFO:teuthology.orchestra.run.smithi072.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-05-09T03:00:16.132 INFO:teuthology.orchestra.run.smithi096.stdout:(3/36): ceph-mds-19.0.0-3603.gc152e9f4.el9.x86_ 2.9 MB/s | 2.4 MB 00:00 2024-05-09T03:00:16.164 INFO:teuthology.orchestra.run.smithi072.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T03:00:16.166 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-09T03:00:16.214 INFO:teuthology.orchestra.run.smithi072.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-05-09T03:00:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-09T03:00:16.223 INFO:teuthology.orchestra.run.smithi096.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3603.gc152 2.9 MB/s | 274 kB 00:00 2024-05-09T03:00:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-09T03:00:16.266 INFO:teuthology.orchestra.run.smithi072.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-05-09T03:00:16.344 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-09T03:00:16.390 INFO:teuthology.orchestra.run.smithi096.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-05-09T03:00:16.417 INFO:teuthology.orchestra.run.smithi072.stdout:(10/36): ceph-mon-19.0.0-3603.gc152e9f4.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-05-09T03:00:16.423 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-09T03:00:16.442 INFO:teuthology.orchestra.run.smithi072.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.2 MB/s | 215 kB 00:00 2024-05-09T03:00:16.471 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-09T03:00:16.506 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-09T03:00:16.535 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-09T03:00:16.559 INFO:teuthology.orchestra.run.smithi072.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-05-09T03:00:16.570 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-09T03:00:16.585 INFO:teuthology.orchestra.run.smithi072.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-05-09T03:00:16.603 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-09T03:00:16.619 INFO:teuthology.orchestra.run.smithi072.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 587 kB/s | 35 kB 00:00 2024-05-09T03:00:16.644 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-09T03:00:16.644 INFO:teuthology.orchestra.run.smithi072.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 864 kB/s | 51 kB 00:00 2024-05-09T03:00:16.657 INFO:teuthology.orchestra.run.smithi096.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-05-09T03:00:16.670 INFO:teuthology.orchestra.run.smithi072.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 821 kB/s | 42 kB 00:00 2024-05-09T03:00:16.678 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-09T03:00:16.699 INFO:teuthology.orchestra.run.smithi096.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-09T03:00:16.704 INFO:teuthology.orchestra.run.smithi072.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 376 kB/s | 22 kB 00:00 2024-05-09T03:00:16.713 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-09T03:00:16.749 INFO:teuthology.orchestra.run.smithi096.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-05-09T03:00:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-09T03:00:16.791 INFO:teuthology.orchestra.run.smithi096.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-05-09T03:00:16.825 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-09T03:00:16.875 INFO:teuthology.orchestra.run.smithi096.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-05-09T03:00:16.950 INFO:teuthology.orchestra.run.smithi096.stdout:(11/36): ceph-mon-19.0.0-3603.gc152e9f4.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-05-09T03:00:16.963 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-09T03:00:17.017 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:17.021 INFO:teuthology.orchestra.run.smithi072.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-05-09T03:00:17.025 INFO:teuthology.orchestra.run.smithi096.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:17.045 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:17.051 INFO:teuthology.orchestra.run.smithi096.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-05-09T03:00:17.051 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-09T03:00:17.084 INFO:teuthology.orchestra.run.smithi096.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 594 kB/s | 35 kB 00:00 2024-05-09T03:00:17.101 INFO:teuthology.orchestra.run.smithi096.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-09T03:00:17.104 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-09T03:00:17.143 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-09T03:00:17.143 INFO:teuthology.orchestra.run.smithi096.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-05-09T03:00:17.155 INFO:teuthology.orchestra.run.smithi072.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-05-09T03:00:17.200 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-09T03:00:17.214 INFO:teuthology.orchestra.run.smithi072.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 893 kB/s | 172 kB 00:00 2024-05-09T03:00:17.255 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-09T03:00:17.314 INFO:teuthology.orchestra.run.smithi072.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-05-09T03:00:17.348 INFO:teuthology.orchestra.run.smithi072.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-05-09T03:00:17.368 INFO:teuthology.orchestra.run.smithi096.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-05-09T03:00:17.416 INFO:teuthology.orchestra.run.smithi072.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 175 kB/s | 18 kB 00:00 2024-05-09T03:00:17.427 INFO:teuthology.orchestra.run.smithi096.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 533 kB/s | 151 kB 00:00 2024-05-09T03:00:17.459 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-09T03:00:17.484 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:17.508 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:17.508 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:17.508 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:00:17.509 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:17.509 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:17.509 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:17.513 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-09T03:00:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-09T03:00:17.610 INFO:teuthology.orchestra.run.smithi096.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 937 kB/s | 172 kB 00:00 2024-05-09T03:00:17.636 INFO:teuthology.orchestra.run.smithi096.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-05-09T03:00:17.689 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-09T03:00:17.701 INFO:teuthology.orchestra.run.smithi072.stdout:(24/36): ceph-osd-19.0.0-3603.gc152e9f4.el9.x86 7.8 MB/s | 17 MB 00:02 2024-05-09T03:00:17.711 INFO:teuthology.orchestra.run.smithi096.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-05-09T03:00:17.727 INFO:teuthology.orchestra.run.smithi072.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 61 kB/s | 23 kB 00:00 2024-05-09T03:00:17.733 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-09T03:00:17.735 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:17.752 INFO:teuthology.orchestra.run.smithi072.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 58 kB/s | 19 kB 00:00 2024-05-09T03:00:17.758 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:17.758 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:17.758 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:00:17.758 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:17.759 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:17.759 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:17.811 INFO:teuthology.orchestra.run.smithi096.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-05-09T03:00:17.822 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:17.844 INFO:teuthology.orchestra.run.smithi072.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 861 kB/s | 79 kB 00:00 2024-05-09T03:00:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-09T03:00:17.878 INFO:teuthology.orchestra.run.smithi072.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-05-09T03:00:17.937 INFO:teuthology.orchestra.run.smithi096.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-05-09T03:00:17.945 INFO:teuthology.orchestra.run.smithi072.stdout:(29/36): python3-jaraco-text-3.2.0-6.el9.noarch 81 kB/s | 20 kB 00:00 2024-05-09T03:00:18.004 INFO:teuthology.orchestra.run.smithi072.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 130 kB/s | 16 kB 00:00 2024-05-09T03:00:18.025 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 31/36 2024-05-09T03:00:18.046 INFO:teuthology.orchestra.run.smithi072.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-05-09T03:00:18.047 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:18.073 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:18.074 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:18.074 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:00:18.074 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:18.074 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:18.074 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:18.079 INFO:teuthology.orchestra.run.smithi096.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 137 kB/s | 19 kB 00:00 2024-05-09T03:00:18.097 INFO:teuthology.orchestra.run.smithi072.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-05-09T03:00:18.139 INFO:teuthology.orchestra.run.smithi072.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 465 kB/s | 90 kB 00:00 2024-05-09T03:00:18.181 INFO:teuthology.orchestra.run.smithi072.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-05-09T03:00:18.240 INFO:teuthology.orchestra.run.smithi072.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 198 kB/s | 20 kB 00:00 2024-05-09T03:00:18.254 INFO:teuthology.orchestra.run.smithi096.stdout:(25/36): ceph-osd-19.0.0-3603.gc152e9f4.el9.x86 7.4 MB/s | 17 MB 00:02 2024-05-09T03:00:18.298 INFO:teuthology.orchestra.run.smithi072.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-05-09T03:00:18.300 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:18.300 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.2 MB/s | 30 MB 00:04 2024-05-09T03:00:18.421 INFO:teuthology.orchestra.run.smithi096.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 278 kB/s | 46 kB 00:00 2024-05-09T03:00:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:18.484 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:18.484 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:18.571 INFO:teuthology.orchestra.run.smithi096.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 526 kB/s | 79 kB 00:00 2024-05-09T03:00:18.688 INFO:teuthology.orchestra.run.smithi096.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-05-09T03:00:18.705 INFO:teuthology.orchestra.run.smithi096.stdout:(29/36): python3-jaraco-text-3.2.0-6.el9.noarch 31 kB/s | 20 kB 00:00 2024-05-09T03:00:18.789 INFO:teuthology.orchestra.run.smithi096.stdout:(30/36): python3-cherrypy-18.6.1-2.el9.noarch.r 304 kB/s | 358 kB 00:01 2024-05-09T03:00:18.814 INFO:teuthology.orchestra.run.smithi096.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-05-09T03:00:18.839 INFO:teuthology.orchestra.run.smithi096.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 672 kB/s | 90 kB 00:00 2024-05-09T03:00:18.865 INFO:teuthology.orchestra.run.smithi096.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-05-09T03:00:18.960 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:18.961 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:18.981 INFO:teuthology.orchestra.run.smithi096.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-05-09T03:00:19.007 INFO:teuthology.orchestra.run.smithi096.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-05-09T03:00:19.048 INFO:teuthology.orchestra.run.smithi096.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2024-05-09T03:00:19.051 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:19.051 INFO:teuthology.orchestra.run.smithi096.stdout:Total 7.5 MB/s | 30 MB 00:04 2024-05-09T03:00:19.191 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:19.231 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:19.231 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:19.534 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 6/36 2024-05-09T03:00:19.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-09T03:00:19.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-09T03:00:19.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-09T03:00:19.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-09T03:00:19.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-09T03:00:19.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-09T03:00:19.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-09T03:00:19.618 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-09T03:00:19.674 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:19.674 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:19.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:20.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:20.233 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:20.255 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:20.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:20.319 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-09T03:00:20.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:20.370 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-09T03:00:20.407 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-09T03:00:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:00:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:00:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:20.480 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:20.481 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-09T03:00:20.524 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-09T03:00:20.567 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-09T03:00:20.660 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-09T03:00:20.740 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-09T03:00:20.789 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-09T03:00:20.825 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-09T03:00:20.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-05-09T03:00:20.865 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-09T03:00:20.899 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-09T03:00:20.933 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-09T03:00:20.934 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:20.973 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-09T03:00:20.993 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-09T03:00:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-09T03:00:21.034 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-09T03:00:21.037 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-09T03:00:21.070 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-09T03:00:21.076 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-09T03:00:21.141 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-09T03:00:21.160 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-09T03:00:21.177 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-09T03:00:21.210 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-09T03:00:21.301 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-09T03:00:21.302 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-09T03:00:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-09T03:00:21.389 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-09T03:00:21.421 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-09T03:00:21.429 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:58 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:21.443 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-09T03:00:21.456 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-09T03:00:21.482 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-09T03:00:21.486 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-09T03:00:21.511 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:21.520 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-09T03:00:21.541 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-09T03:00:21.552 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:21.553 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:00:21.553 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:21.553 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-09T03:00:21.594 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-09T03:00:21.604 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-09T03:00:21.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-05-09T03:00:21.629 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-09T03:00:21.663 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-09T03:00:21.701 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-09T03:00:21.774 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-09T03:00:21.819 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-09T03:00:21.875 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-09T03:00:21.919 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-09T03:00:21.954 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-09T03:00:22.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-09T03:00:22.043 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-09T03:00:22.054 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-09T03:00:22.093 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-09T03:00:22.094 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-09T03:00:22.149 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 6/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-09T03:00:22.153 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-09T03:00:22.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-09T03:00:22.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-09T03:00:22.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-09T03:00:22.179 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:59 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:22.214 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-09T03:00:22.228 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-09T03:00:22.302 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 764 k 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 764 k 2024-05-09T03:00:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 770 k 2024-05-09T03:00:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:22.385 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 31/36 2024-05-09T03:00:22.404 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-09T03:00:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:22.442 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:22.471 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-09T03:00:22.541 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-09T03:00:22.584 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-19.0.0-3603.gc152e9f4.el9.noarch.rpm 2.7 MB/s | 764 kB 00:00 2024-05-09T03:00:22.585 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:22.585 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.6 MB/s | 764 kB 00:00 2024-05-09T03:00:22.585 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:22.590 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:22.590 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:22.596 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:22.597 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:22.638 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-09T03:00:22.674 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-09T03:00:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-09T03:00:22.967 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 31/36 2024-05-09T03:00:22.998 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:23.024 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 32/36 2024-05-09T03:00:23.024 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:23.025 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:00:23.025 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:23.025 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:00:23.025 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:23.102 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:00:23.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:00:23.104 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:00:23.105 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:00:23.105 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:00:23.105 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:23.105 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:23.145 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:23.230 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:23.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-05-09T03:00:23.610 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:23.841 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:23.841 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:23.841 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:23.842 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:23.842 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:23.842 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:23.922 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:23.922 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:23.922 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:00:23.922 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:23.923 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:23.923 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:24.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-05-09T03:00:24.045 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:58 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:24.128 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:24.169 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:24.170 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:00:24.170 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:24.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-05-09T03:00:24.362 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:24.387 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:24.457 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:24.484 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 33/36 2024-05-09T03:00:24.484 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:24.485 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:00:24.485 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:24.485 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:00:24.485 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:24.614 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:24.616 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:01 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:24.637 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:24.694 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:24.743 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:24.743 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:24.743 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-09T03:00:24.743 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:24.743 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 145 k 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 145 k 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 435 k 2024-05-09T03:00:24.744 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:24.815 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:58 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:24.933 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:24.938 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 764 k 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 764 k 2024-05-09T03:00:24.939 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 770 k 2024-05-09T03:00:24.940 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:24.956 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 34/36 2024-05-09T03:00:24.956 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:24.957 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:00:24.957 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:24.957 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:00:24.957 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:25.174 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-19.0.0-3603.gc152e9 334 kB/s | 145 kB 00:00 2024-05-09T03:00:25.174 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:25.175 INFO:teuthology.orchestra.run.smithi195.stdout:Total 333 kB/s | 145 kB 00:00 2024-05-09T03:00:25.175 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:25.182 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:25.183 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:25.183 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:25.207 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 35/36 2024-05-09T03:00:25.207 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:25.207 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:00:25.208 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:25.208 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:00:25.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:25.216 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.0.0-3603.gc152e9f4.el9.noarch.rpm 2.7 MB/s | 764 kB 00:00 2024-05-09T03:00:25.217 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:25.217 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.7 MB/s | 764 kB 00:00 2024-05-09T03:00:25.217 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:25.222 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:25.222 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:25.228 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:25.229 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:25.239 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:25.240 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:25.271 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:25.304 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:25.475 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:25.500 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:25.500 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:25.500 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:00:25.500 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:25.693 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:25.779 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:26.117 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:26.365 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:26.415 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:26.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-05-09T03:00:26.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 6/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-09T03:00:27.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-09T03:00:27.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-09T03:00:27.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-09T03:00:27.175 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:01 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:27.206 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:04 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:27.239 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 36/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 3/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 4/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 5/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9. 6/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-09T03:00:27.240 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-09T03:00:27.241 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-09T03:00:27.242 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-09T03:00:27.242 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-09T03:00:27.243 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-09T03:00:27.243 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-09T03:00:27.243 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-09T03:00:27.243 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-09T03:00:27.285 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:27.304 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 145 k 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:00:27.305 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:27.306 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-05-09T03:00:27.306 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 435 k 2024-05-09T03:00:27.306 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:27.328 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:27.330 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:00:27.330 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:27.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-05-09T03:00:27.656 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.0.0-3603.gc152e9 414 kB/s | 145 kB 00:00 2024-05-09T03:00:27.656 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:27.657 INFO:teuthology.orchestra.run.smithi181.stdout:Total 412 kB/s | 145 kB 00:00 2024-05-09T03:00:27.657 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:27.664 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:27.664 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:27.721 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:27.721 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:27.864 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:27.915 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:27.940 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:27.941 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:27.941 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:00:27.941 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:27.959 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:04 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:00:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:00:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:28.070 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 3.5 M 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 34 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 16 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:28.088 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:28.089 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-05-09T03:00:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:28.089 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 8.6 M 2024-05-09T03:00:28.089 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 104 M 2024-05-09T03:00:28.089 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:28.148 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-09T03:00:28.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:28.148 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:00:28.148 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.148 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:00:28.149 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:00:28.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:28.151 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:28.428 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-05-09T03:00:28.604 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-base 2024-05-09T03:00:28.787 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3603.gc15 85 kB/s | 16 kB 00:00 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:28.873 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:28.895 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3603.gc1 113 kB/s | 34 kB 00:00 2024-05-09T03:00:29.005 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:57 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:29.045 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3603.gc152e9f 7.8 MB/s | 3.5 MB 00:00 2024-05-09T03:00:29.087 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:29.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-05-09T03:00:29.128 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:29.129 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:00:29.129 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:29.179 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:02 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:29.200 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-05-09T03:00:29.262 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-base-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:29.279 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-05-09T03:00:29.303 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:29.304 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:00:29.305 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:29.389 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install cephadm 2024-05-09T03:00:29.688 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:03 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:29.768 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:57 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:29.771 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:29.796 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-05-09T03:00:29.813 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:29.814 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:00:29.814 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:29.821 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): python3-protobuf-3.14.0-13.el9.noarch.r 291 kB/s | 269 kB 00:00 2024-05-09T03:00:29.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 764 k 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:29.891 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:00:29.892 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:29.892 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 764 k 2024-05-09T03:00:29.892 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 770 k 2024-05-09T03:00:29.892 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:29.954 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:02 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:30.077 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 764 k 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 764 k 2024-05-09T03:00:30.079 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 770 k 2024-05-09T03:00:30.079 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:00:30.096 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-05-09T03:00:30.167 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-19.0.0-3603.gc152e9f4.el9.noarch.rpm 2.7 MB/s | 764 kB 00:00 2024-05-09T03:00:30.167 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:30.168 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.7 MB/s | 764 kB 00:00 2024-05-09T03:00:30.168 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:30.173 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:30.173 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:30.179 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:30.179 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:30.256 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:30.388 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 495 kB/s | 548 kB 00:01 2024-05-09T03:00:30.459 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:04 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:30.495 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm-19.0.0-3603.gc152e9f4.el9.noarch.rpm 1.8 MB/s | 764 kB 00:00 2024-05-09T03:00:30.495 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:30.496 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.8 MB/s | 764 kB 00:00 2024-05-09T03:00:30.496 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:30.501 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:30.501 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:30.507 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:30.507 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:30.522 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 230 kB/s | 31 kB 00:00 2024-05-09T03:00:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 304 kB/s | 144 kB 00:00 2024-05-09T03:00:30.578 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:30.584 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 3.5 M 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 34 k 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 16 k 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:00:30.586 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-05-09T03:00:30.587 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:30.588 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 8.6 M 2024-05-09T03:00:30.588 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 104 M 2024-05-09T03:00:30.588 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:30.787 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:30.814 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 646 kB/s | 188 kB 00:00 2024-05-09T03:00:31.060 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.146 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:31.383 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:31.554 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.604 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-05-09T03:00:31.751 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3603.gc1 87 kB/s | 34 kB 00:00 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/1 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:31.794 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:31.843 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3603.gc15 34 kB/s | 16 kB 00:00 2024-05-09T03:00:32.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-immutable-object-cache 2024-05-09T03:00:32.175 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:00 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:32.300 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:32.300 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:32.300 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 145 k 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 145 k 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 435 k 2024-05-09T03:00:32.301 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:32.452 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 443 kB/s | 269 kB 00:00 2024-05-09T03:00:32.519 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): ceph-mgr-dashboard-19.0.0-3603.gc152e9f 3.1 MB/s | 3.5 MB 00:01 2024-05-09T03:00:32.601 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:05 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:32.694 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-05-09T03:00:32.726 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 145 k 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:00:32.727 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:32.728 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 145 k 2024-05-09T03:00:32.728 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 435 k 2024-05-09T03:00:32.728 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:00:32.789 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-19.0.0-3603.gc152e9 296 kB/s | 145 kB 00:00 2024-05-09T03:00:32.790 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:32.790 INFO:teuthology.orchestra.run.smithi096.stdout:Total 296 kB/s | 145 kB 00:00 2024-05-09T03:00:32.790 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:32.798 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:32.798 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:32.856 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:32.856 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:32.986 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-09T03:00:33.000 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:33.028 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-immutable-object-cache-19.0.0-3603.gc152e9 483 kB/s | 145 kB 00:00 2024-05-09T03:00:33.029 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:33.029 INFO:teuthology.orchestra.run.smithi072.stdout:Total 480 kB/s | 145 kB 00:00 2024-05-09T03:00:33.029 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:33.037 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:33.037 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:33.091 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:33.091 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:00:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:33.092 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:33.093 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:33.219 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 782 kB/s | 548 kB 00:00 2024-05-09T03:00:33.230 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 612 kB/s | 2.0 MB 00:03 2024-05-09T03:00:33.233 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:33.233 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.7 MB/s | 8.6 MB 00:05 2024-05-09T03:00:33.238 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:33.286 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:33.305 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:33.305 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:33.341 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:33.366 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:33.366 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:00:33.366 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:00:33.366 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:33.484 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:33.484 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:33.586 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 761 kB/s | 863 kB 00:01 2024-05-09T03:00:33.620 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 361 kB/s | 144 kB 00:00 2024-05-09T03:00:33.845 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 119 kB/s | 31 kB 00:00 2024-05-09T03:00:34.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:34.023 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:34.023 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:34.024 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:34.024 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:34.024 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:34.081 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:34.158 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 351 kB/s | 188 kB 00:00 2024-05-09T03:00:34.166 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-09T03:00:34.209 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-09T03:00:34.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-05-09T03:00:34.255 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-09T03:00:34.294 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-09T03:00:34.306 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4. 1/1 2024-05-09T03:00:34.306 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:34.306 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:00:34.307 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:00:34.307 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:34.307 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:34.354 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.5 MB/s | 2.0 MB 00:01 2024-05-09T03:00:34.357 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:34.358 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.3 MB/s | 8.6 MB 00:03 2024-05-09T03:00:34.397 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-09T03:00:34.411 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:34.429 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:34.429 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:34.529 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-09T03:00:34.547 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr 2024-05-09T03:00:34.604 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:34.604 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:34.630 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-09T03:00:34.682 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-09T03:00:34.710 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:34.751 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 10/12 2024-05-09T03:00:34.826 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:02 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:34.908 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:34.950 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:34.951 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:00:34.951 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:35.024 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-05-09T03:00:35.124 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:08 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:35.206 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-mgr-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:00:35.218 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:35.249 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:35.250 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:00:35.250 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:35.305 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-09T03:00:35.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-dashboard 2024-05-09T03:00:35.340 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-09T03:00:35.386 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-09T03:00:35.425 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-09T03:00:35.530 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-09T03:00:35.588 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:03 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:35.661 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-09T03:00:35.711 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:35.712 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:35.712 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 3.5 M 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 34 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 16 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:00:35.713 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-05-09T03:00:35.714 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:35.715 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 8.6 M 2024-05-09T03:00:35.715 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 104 M 2024-05-09T03:00:35.715 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:35.755 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-09T03:00:35.798 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-09T03:00:35.799 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 11/12 2024-05-09T03:00:35.827 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:35.834 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:35.851 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 10/12 2024-05-09T03:00:35.887 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:08 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:36.012 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 3.5 M 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 34 k 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 16 k 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:00:36.013 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-05-09T03:00:36.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:36.015 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 8.6 M 2024-05-09T03:00:36.015 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 104 M 2024-05-09T03:00:36.015 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:00:36.703 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3603.gc1 88 kB/s | 34 kB 00:00 2024-05-09T03:00:36.753 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3603.gc15 38 kB/s | 16 kB 00:00 2024-05-09T03:00:36.896 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 11/12 2024-05-09T03:00:36.943 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:37.028 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 979 kB/s | 269 kB 00:00 2024-05-09T03:00:37.081 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3603.gc1 117 kB/s | 34 kB 00:00 2024-05-09T03:00:37.107 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3603.gc15 52 kB/s | 16 kB 00:00 2024-05-09T03:00:37.120 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-05-09T03:00:37.207 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:37.207 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 1/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 2/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 3/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-09T03:00:37.208 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-09T03:00:37.458 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3603.gc152e9f 5.3 MB/s | 3.5 MB 00:00 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:00:37.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:37.523 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-05-09T03:00:37.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-09T03:00:37.795 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-05-09T03:00:37.835 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-05-09T03:00:37.944 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 322 kB/s | 269 kB 00:00 2024-05-09T03:00:38.071 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): ceph-mgr-dashboard-19.0.0-3603.gc152e9f 2.0 MB/s | 3.5 MB 00:01 2024-05-09T03:00:38.096 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 808 kB/s | 863 kB 00:01 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 1/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 2/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 3/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-09T03:00:38.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-09T03:00:38.135 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:38.135 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-09T03:00:38.135 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-09T03:00:38.220 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 70 kB/s | 19 kB 00:00 2024-05-09T03:00:38.238 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 4.6 MB/s | 2.0 MB 00:00 2024-05-09T03:00:38.337 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 983 kB/s | 863 kB 00:00 2024-05-09T03:00:38.347 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-05-09T03:00:38.353 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:15 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:38.422 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:38.463 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 84 kB/s | 31 kB 00:00 2024-05-09T03:00:38.485 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:38.486 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-09T03:00:38.486 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:38.486 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-09T03:00:38.486 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 7.4 M 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:00:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 38 M 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 209 M 2024-05-09T03:00:38.488 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:38.589 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 279 kB/s | 144 kB 00:00 2024-05-09T03:00:38.593 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:38.593 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.0 MB/s | 8.6 MB 00:02 2024-05-09T03:00:38.646 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:38.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-09T03:00:38.663 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:38.663 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:38.842 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:38.842 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:38.846 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 542 kB/s | 548 kB 00:01 2024-05-09T03:00:39.139 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-36 13 MB/s | 7.4 MB 00:00 2024-05-09T03:00:39.164 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 326 kB/s | 195 kB 00:00 2024-05-09T03:00:39.239 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:13 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:39.371 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:39.372 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-05-09T03:00:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:39.372 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-05-09T03:00:39.372 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 7.4 M 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:00:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-05-09T03:00:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:39.375 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-05-09T03:00:39.375 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 209 M 2024-05-09T03:00:39.375 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:39.384 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 982 kB/s | 803 kB 00:00 2024-05-09T03:00:39.447 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-09T03:00:39.575 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-09T03:00:39.621 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-09T03:00:39.652 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-09T03:00:39.754 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-09T03:00:39.843 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 47 kB/s | 33 kB 00:00 2024-05-09T03:00:39.879 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-09T03:00:39.957 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 600 kB/s | 195 kB 00:00 2024-05-09T03:00:39.997 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-09T03:00:40.007 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 660 kB/s | 33 kB 00:00 2024-05-09T03:00:40.034 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 28 kB/s | 18 kB 00:00 2024-05-09T03:00:40.038 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 1.1 MB/s | 2.0 MB 00:01 2024-05-09T03:00:40.048 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-09T03:00:40.076 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:40.108 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 10/12 2024-05-09T03:00:40.174 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): ceph-mgr-diskprediction-local-19.0.0-36 14 MB/s | 7.4 MB 00:00 2024-05-09T03:00:40.180 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 78 kB/s | 144 kB 00:01 2024-05-09T03:00:40.205 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 23 kB/s | 31 kB 00:01 2024-05-09T03:00:40.224 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-05-09T03:00:40.283 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-05-09T03:00:40.316 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-05-09T03:00:40.380 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 551 kB/s | 188 kB 00:00 2024-05-09T03:00:40.385 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:40.385 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.0 MB/s | 8.6 MB 00:04 2024-05-09T03:00:40.400 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-09T03:00:40.439 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:40.457 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:40.457 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:40.467 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.5 MB/s | 3.0 MB 00:00 2024-05-09T03:00:40.542 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-05-09T03:00:40.637 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:40.637 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:40.692 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 41 kB/s | 35 kB 00:00 2024-05-09T03:00:40.743 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.9 MB/s | 3.0 MB 00:01 2024-05-09T03:00:40.776 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 9.6 MB/s | 4.7 MB 00:00 2024-05-09T03:00:40.892 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-05-09T03:00:41.163 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 11/12 2024-05-09T03:00:41.192 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:41.246 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:41.268 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-05-09T03:00:41.269 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:41.269 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 38 MB 00:01 2024-05-09T03:00:41.332 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-09T03:00:41.376 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-09T03:00:41.425 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-09T03:00:41.465 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-09T03:00:41.586 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-09T03:00:41.626 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:41.653 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:41.653 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:41.720 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-09T03:00:41.743 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 2.8 MB/s | 4.7 MB 00:01 2024-05-09T03:00:41.768 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 227 kB/s | 244 kB 00:01 2024-05-09T03:00:41.814 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-09T03:00:41.860 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-09T03:00:41.897 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:41.930 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 10/12 2024-05-09T03:00:42.074 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:42.074 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:42.327 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 1/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 2/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 3/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-09T03:00:42.498 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-09T03:00:42.499 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-09T03:00:42.499 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-09T03:00:42.499 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:42.499 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-09T03:00:42.499 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-09T03:00:42.777 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 242 kB/s | 250 kB 00:01 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:00:42.815 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:42.816 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:42.844 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:42.912 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-09T03:00:42.966 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:42.970 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 11/12 2024-05-09T03:00:43.009 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:43.061 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-09T03:00:43.066 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:43.101 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-09T03:00:43.279 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-09T03:00:43.316 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-09T03:00:43.394 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-09T03:00:43.630 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:11 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:43.752 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:43.753 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-09T03:00:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:43.753 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-09T03:00:43.753 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 7.4 M 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:00:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 38 M 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 209 M 2024-05-09T03:00:43.755 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:43.760 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.2 MB/s | 16 MB 00:01 2024-05-09T03:00:43.762 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:43.763 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.2 MB/s | 38 MB 00:05 2024-05-09T03:00:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-09T03:00:44.120 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:44.147 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:44.147 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:44.170 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-09T03:00:44.581 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:44.581 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:44.938 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 340 kB/s | 195 kB 00:00 2024-05-09T03:00:45.096 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 33 kB 00:00 2024-05-09T03:00:45.126 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 12/12 2024-05-09T03:00:45.126 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el 1/12 2024-05-09T03:00:45.126 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noa 2/12 2024-05-09T03:00:45.126 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9 3/12 2024-05-09T03:00:45.126 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-09T03:00:45.127 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-09T03:00:45.196 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): ceph-mgr-diskprediction-local-19.0.0-36 8.9 MB/s | 7.4 MB 00:00 2024-05-09T03:00:45.297 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 182 kB/s | 18 kB 00:00 2024-05-09T03:00:45.322 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): libgfortran-11.4.1-3.el9.x86_64.rpm 837 kB/s | 803 kB 00:00 2024-05-09T03:00:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:45.388 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 384 kB/s | 35 kB 00:00 2024-05-09T03:00:45.391 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:45.401 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-09T03:00:45.430 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-09T03:00:45.430 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:45.430 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:00:45.430 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:45.431 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:00:45.450 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:45.539 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-05-09T03:00:45.550 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:45.593 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-09T03:00:45.716 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-09T03:00:45.770 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-09T03:00:45.808 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-09T03:00:45.885 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-09T03:00:46.081 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-05-09T03:00:46.198 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-05-09T03:00:46.284 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:19 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:00:46.404 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 7.4 M 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:00:46.406 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-05-09T03:00:46.407 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:00:46.408 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 38 M 2024-05-09T03:00:46.408 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 209 M 2024-05-09T03:00:46.408 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:00:46.423 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-09T03:00:46.441 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:46.547 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-09T03:00:46.664 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-09T03:00:47.089 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.3 MB/s | 5.1 MB 00:01 2024-05-09T03:00:47.221 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 724 kB/s | 195 kB 00:00 2024-05-09T03:00:47.264 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.4 MB/s | 4.7 MB 00:01 2024-05-09T03:00:47.563 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): ceph-mgr-diskprediction-local-19.0.0-36 12 MB/s | 7.4 MB 00:00 2024-05-09T03:00:47.672 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-05-09T03:00:47.910 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:48.014 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 41 kB/s | 33 kB 00:00 2024-05-09T03:00:48.292 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 29 kB/s | 18 kB 00:00 2024-05-09T03:00:48.733 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 49 kB/s | 35 kB 00:00 2024-05-09T03:00:48.967 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-05-09T03:00:48.971 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-09T03:00:49.017 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 1/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-09T03:00:49.028 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:49.029 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:00:49.408 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:49.409 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:49.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-05-09T03:00:49.759 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.3 MB/s | 4.7 MB 00:01 2024-05-09T03:00:49.793 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 230 kB/s | 244 kB 00:01 2024-05-09T03:00:50.140 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.1 MB/s | 16 MB 00:03 2024-05-09T03:00:50.140 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:50.141 INFO:teuthology.orchestra.run.smithi096.stdout:Total 6.0 MB/s | 38 MB 00:06 2024-05-09T03:00:50.360 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:50.452 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-05-09T03:00:50.484 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 49 k 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:00:50.486 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:50.487 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:50.488 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-05-09T03:00:50.488 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:50.488 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-05-09T03:00:50.488 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-05-09T03:00:50.488 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:50.503 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:00:50.530 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:00:50.530 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:00:50.819 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 237 kB/s | 250 kB 00:01 2024-05-09T03:00:50.942 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 19 kB 00:00 2024-05-09T03:00:50.965 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:00:50.965 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:00:50.967 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 105 kB/s | 26 kB 00:00 2024-05-09T03:00:51.143 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-19.0.0-3603.gc152e9f4.el9 118 kB/s | 49 kB 00:00 2024-05-09T03:00:51.310 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-05-09T03:00:51.343 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 554 kB/s | 222 kB 00:00 2024-05-09T03:00:51.460 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 356 kB/s | 54 kB 00:00 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 1/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-09T03:00:51.585 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-09T03:00:51.586 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-09T03:00:51.586 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-09T03:00:51.586 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:51.586 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-09T03:00:51.661 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 102 kB/s | 32 kB 00:00 2024-05-09T03:00:51.703 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 498 kB/s | 279 kB 00:00 2024-05-09T03:00:51.720 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-05-09T03:00:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:00:51.795 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-09T03:00:51.811 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.1 MB/s | 16 MB 00:02 2024-05-09T03:00:51.813 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:51.813 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.0 MB/s | 38 MB 00:05 2024-05-09T03:00:51.837 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:51.938 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:00:51.974 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:51.975 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:51.980 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-09T03:00:52.028 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 560 kB/s | 206 kB 00:00 2024-05-09T03:00:52.078 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-05-09T03:00:52.157 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-09T03:00:52.170 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2024-05-09T03:00:52.171 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:00:52.195 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-09T03:00:52.197 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:00:52.197 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:00:52.262 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-05-09T03:00:52.262 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:52.263 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-05-09T03:00:52.273 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-09T03:00:52.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-05-09T03:00:52.335 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:52.351 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:52.351 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:52.439 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:52.439 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:52.632 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:00:52.632 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:00:52.901 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:29 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:52.926 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-09T03:00:52.941 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:53.024 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 49 k 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:00:53.025 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:53.026 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-05-09T03:00:53.027 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:53.027 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.2 M 2024-05-09T03:00:53.027 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-05-09T03:00:53.027 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:53.050 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-09T03:00:53.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-09T03:00:53.163 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-09T03:00:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-09T03:00:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-09T03:00:53.297 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-09T03:00:53.380 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:00:53.391 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-09T03:00:53.447 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-09T03:00:53.484 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): ceph-mgr-rook-19.0.0-3603.gc152e9f4.el9 261 kB/s | 49 kB 00:00 2024-05-09T03:00:53.494 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-09T03:00:53.509 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 120 kB/s | 26 kB 00:00 2024-05-09T03:00:53.534 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 80 kB/s | 19 kB 00:00 2024-05-09T03:00:53.574 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-09T03:00:53.596 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:53.640 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-09T03:00:53.817 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-09T03:00:53.848 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-09T03:00:53.876 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 465 kB/s | 159 kB 00:00 2024-05-09T03:00:53.901 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 532 kB/s | 222 kB 00:00 2024-05-09T03:00:53.927 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-09T03:00:54.026 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 357 kB/s | 54 kB 00:00 2024-05-09T03:00:54.135 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 464 kB/s | 279 kB 00:00 2024-05-09T03:00:54.283 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:54.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-09T03:00:54.346 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-09T03:00:54.380 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-09T03:00:54.394 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-05-09T03:00:54.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-09T03:00:54.450 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:00:54.544 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 30 kB/s | 15 kB 00:00 2024-05-09T03:00:54.610 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-09T03:00:54.753 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-09T03:00:54.852 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 192 kB/s | 59 kB 00:00 2024-05-09T03:00:55.061 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 222 kB/s | 206 kB 00:00 2024-05-09T03:00:55.178 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 276 kB/s | 90 kB 00:00 2024-05-09T03:00:55.212 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2024-05-09T03:00:55.215 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:55.215 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.0 MB/s | 2.2 MB 00:02 2024-05-09T03:00:55.257 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 1/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-09T03:00:55.258 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-09T03:00:55.259 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-09T03:00:55.282 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:00:55.297 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:00:55.298 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:00:55.327 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-09T03:00:55.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:55.382 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:00:55.382 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:00:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:55.581 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:00:55.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-05-09T03:00:55.882 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:00:56.015 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:56.041 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-09T03:00:56.112 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-09T03:00:56.176 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-09T03:00:56.205 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-09T03:00:56.247 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-09T03:00:56.357 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-09T03:00:56.424 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:30 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:00:56.466 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-09T03:00:56.531 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-09T03:00:56.545 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:00:56.546 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 142 k 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:00:56.547 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:00:56.548 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-05-09T03:00:56.548 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:00:56.548 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-05-09T03:00:56.548 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-05-09T03:00:56.548 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:00:57.101 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-09T03:00:57.138 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-09T03:00:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-09T03:00:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-09T03:00:57.365 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3603.gc152e9f4.e 243 kB/s | 142 kB 00:00 2024-05-09T03:00:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-09T03:00:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:00:57.948 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 940 kB/s | 548 kB 00:00 2024-05-09T03:00:58.057 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 533 kB/s | 58 kB 00:00 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 1/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:00:58.099 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-09T03:00:58.174 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 436 kB/s | 51 kB 00:00 2024-05-09T03:00:58.208 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 1/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-09T03:00:58.209 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-09T03:00:58.232 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 170 kB/s | 248 kB 00:01 2024-05-09T03:00:58.491 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.5 MB/s | 6.0 MB 00:01 2024-05-09T03:00:58.493 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:00:58.493 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-05-09T03:00:58.518 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-09T03:00:58.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:58.518 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:00:58.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:00:58.520 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:58.520 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:00:58.556 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:58.557 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:00:58.566 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:00:58.577 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:00:58.577 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:00:58.674 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:00:58.674 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:00:58.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-05-09T03:00:58.899 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-05-09T03:00:58.979 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:00:59.138 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-09T03:00:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-09T03:00:59.350 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:36 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:00:59.473 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:27 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:00:59.485 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:00:59.486 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:59.486 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:00:59.486 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 142 k 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-05-09T03:00:59.487 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:00:59.488 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 7.0 M 2024-05-09T03:00:59.488 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 33 M 2024-05-09T03:00:59.488 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:00:59.596 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 49 k 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:00:59.597 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:00:59.598 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:00:59.599 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-05-09T03:00:59.599 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:00:59.599 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.2 M 2024-05-09T03:00:59.599 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-05-09T03:00:59.599 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:00:59.696 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-09T03:00:59.796 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-09T03:00:59.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-09T03:00:59.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:00:59.962 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): ceph-mgr-rook-19.0.0-3603.gc152e9f4.el9 198 kB/s | 49 kB 00:00 2024-05-09T03:00:59.987 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 93 kB/s | 26 kB 00:00 2024-05-09T03:01:00.012 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 64 kB/s | 19 kB 00:00 2024-05-09T03:01:00.490 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 438 kB/s | 248 kB 00:00 2024-05-09T03:01:00.515 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3603.gc152e9f4.e 241 kB/s | 142 kB 00:00 2024-05-09T03:01:00.587 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 276 kB/s | 159 kB 00:00 2024-05-09T03:01:00.620 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 337 kB/s | 222 kB 00:00 2024-05-09T03:01:00.622 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:00.622 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/6 2024-05-09T03:01:00.622 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-09T03:01:00.622 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-09T03:01:00.622 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-09T03:01:00.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-09T03:01:00.679 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 552 kB/s | 32 kB 00:00 2024-05-09T03:01:00.713 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 450 kB/s | 15 kB 00:00 2024-05-09T03:01:00.771 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-google-auth-2.29.0-1.el9.noarch 3.4 MB/s | 206 kB 00:00 2024-05-09T03:01:00.813 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 348 kB/s | 279 kB 00:00 2024-05-09T03:01:00.847 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 14 MB/s | 1.0 MB 00:00 2024-05-09T03:01:00.880 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-05-09T03:01:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-09T03:01:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:00.901 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:00.902 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:00.906 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-05-09T03:01:00.940 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2024-05-09T03:01:00.947 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 149 kB/s | 54 kB 00:00 2024-05-09T03:01:00.948 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:00.949 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-05-09T03:01:01.020 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:01.035 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:01.035 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:01.115 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2024-05-09T03:01:01.120 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:01.122 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:01.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-05-09T03:01:01.207 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 764 kB/s | 548 kB 00:00 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 12/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e 1/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-09T03:01:01.257 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-09T03:01:01.612 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:01:01.668 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:01.669 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:01.726 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:01.777 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-09T03:01:01.832 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-09T03:01:01.849 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:01.849 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:01.849 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 891 k 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-05-09T03:01:01.850 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:01.851 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 971 k 2024-05-09T03:01:01.851 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.8 M 2024-05-09T03:01:01.851 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:01.913 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-09T03:01:01.950 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-09T03:01:01.991 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-09T03:01:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-09T03:01:02.177 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): ceph-fuse-19.0.0-3603.gc152e9f4.el9.x86_ 3.1 MB/s | 891 kB 00:00 2024-05-09T03:01:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-09T03:01:02.243 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-09T03:01:02.258 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.6 MB/s | 6.0 MB 00:02 2024-05-09T03:01:02.261 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:02.261 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.5 MB/s | 7.0 MB 00:02 2024-05-09T03:01:02.287 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-rook 2024-05-09T03:01:02.335 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:02.347 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:02.347 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:02.445 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:02.445 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:02.669 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 104 kB/s | 80 kB 00:00 2024-05-09T03:01:02.670 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:02.670 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.2 MB/s | 971 kB 00:00 2024-05-09T03:01:02.683 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:02.705 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:02.705 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:02.745 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:02.784 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:02.784 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:02.865 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:35 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:02.926 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-09T03:01:02.953 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-09T03:01:02.965 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:02.985 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-09T03:01:02.987 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 49 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:01:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout:Install 13 Packages 2024-05-09T03:01:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:02.991 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.2 M 2024-05-09T03:01:02.991 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2024-05-09T03:01:02.991 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:02.999 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-09T03:01:03.032 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-09T03:01:03.081 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-09T03:01:03.110 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:01:03.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-09T03:01:03.176 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:03.514 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-09T03:01:03.614 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-09T03:01:03.684 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-09T03:01:03.713 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:03.753 INFO:teuthology.orchestra.run.smithi072.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 147 kB/s | 19 kB 00:00 2024-05-09T03:01:03.788 INFO:teuthology.orchestra.run.smithi072.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 26 kB 00:00 2024-05-09T03:01:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:03.838 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/2 2024-05-09T03:01:03.839 INFO:teuthology.orchestra.run.smithi072.stdout:(3/13): ceph-mgr-rook-19.0.0-3603.gc152e9f4.el9 228 kB/s | 49 kB 00:00 2024-05-09T03:01:03.869 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:01:03.869 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 1/13 2024-05-09T03:01:03.869 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-09T03:01:03.870 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-09T03:01:03.948 INFO:teuthology.orchestra.run.smithi072.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-05-09T03:01:03.973 INFO:teuthology.orchestra.run.smithi072.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 859 kB/s | 159 kB 00:00 2024-05-09T03:01:04.007 INFO:teuthology.orchestra.run.smithi072.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 908 kB/s | 54 kB 00:00 2024-05-09T03:01:04.074 INFO:teuthology.orchestra.run.smithi072.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:04.118 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:04.183 INFO:teuthology.orchestra.run.smithi072.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-05-09T03:01:04.194 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-09T03:01:04.194 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:04.194 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:04.195 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:04.209 INFO:teuthology.orchestra.run.smithi072.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 84 kB/s | 15 kB 00:00 2024-05-09T03:01:04.242 INFO:teuthology.orchestra.run.smithi072.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.4 MB/s | 206 kB 00:00 2024-05-09T03:01:04.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-05-09T03:01:04.409 INFO:teuthology.orchestra.run.smithi072.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-05-09T03:01:04.435 INFO:teuthology.orchestra.run.smithi072.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 261 kB/s | 59 kB 00:00 2024-05-09T03:01:04.459 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:04.459 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/6 2024-05-09T03:01:04.459 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-09T03:01:04.459 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-09T03:01:04.460 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-09T03:01:04.460 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-09T03:01:04.460 INFO:teuthology.orchestra.run.smithi072.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 413 kB/s | 90 kB 00:00 2024-05-09T03:01:04.462 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:04.463 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-05-09T03:01:04.470 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-05-09T03:01:04.527 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:04.541 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:04.627 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:04.627 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:04.742 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:04.978 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:05.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-05-09T03:01:05.038 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:33 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:05.106 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:05.106 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:05.106 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 273 k 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-05-09T03:01:05.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:05.108 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 350 k 2024-05-09T03:01:05.108 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.5 M 2024-05-09T03:01:05.108 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:05.140 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:05.160 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 142 k 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:01:05.161 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:05.162 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 7.0 M 2024-05-09T03:01:05.163 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-05-09T03:01:05.163 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:05.316 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-09T03:01:05.398 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-09T03:01:05.463 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-09T03:01:05.501 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-09T03:01:05.542 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-09T03:01:05.598 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:42 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:05.636 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-09T03:01:05.721 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 891 k 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:01:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 971 k 2024-05-09T03:01:05.723 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.8 M 2024-05-09T03:01:05.724 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:05.748 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-09T03:01:05.807 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-09T03:01:06.401 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 237 kB/s | 80 kB 00:00 2024-05-09T03:01:06.526 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-09T03:01:06.566 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 862 kB/s | 77 kB 00:00 2024-05-09T03:01:06.584 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-09T03:01:06.618 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-09T03:01:06.643 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-fuse-19.0.0-3603.gc152e9f4.el9.x86_ 1.5 MB/s | 891 kB 00:00 2024-05-09T03:01:06.644 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:06.644 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.0 MB/s | 971 kB 00:00 2024-05-09T03:01:06.657 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:06.667 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-09T03:01:06.679 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:06.680 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:06.697 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:01:06.741 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-volume-19.0.0-3603.gc152e9f4.el9.no 1.0 MB/s | 273 kB 00:00 2024-05-09T03:01:06.741 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:06.742 INFO:teuthology.orchestra.run.smithi181.stdout:Total 214 kB/s | 350 kB 00:01 2024-05-09T03:01:06.755 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:06.760 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:06.760 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:06.767 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:06.767 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:06.801 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:06.801 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:06.944 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:06.958 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:07.104 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-09T03:01:07.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-09T03:01:07.160 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:07.176 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:07.176 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:07.176 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:01:07.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:07.184 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 13/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 1/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-09T03:01:07.701 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-09T03:01:07.702 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-09T03:01:07.825 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:07.825 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/2 2024-05-09T03:01:07.910 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 1/2 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:01:08.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:01:08.013 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:01:08.013 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:01:08.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:08.013 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-09T03:01:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:08.059 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-09T03:01:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:08.059 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:08.116 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3603.gc152e9f4.e 337 kB/s | 142 kB 00:00 2024-05-09T03:01:08.202 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:08.203 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:08.266 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 433 kB/s | 248 kB 00:00 2024-05-09T03:01:08.315 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-cephadm 2024-05-09T03:01:08.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-volume 2024-05-09T03:01:08.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-05-09T03:01:08.717 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2024-05-09T03:01:08.850 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 747 kB/s | 548 kB 00:00 2024-05-09T03:01:08.876 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-05-09T03:01:08.887 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:41 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:08.927 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:45 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:09.009 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 142 k 2024-05-09T03:01:09.010 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2024-05-09T03:01:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:09.012 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 7.0 M 2024-05-09T03:01:09.012 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 33 M 2024-05-09T03:01:09.012 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:09.051 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 273 k 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:09.052 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 350 k 2024-05-09T03:01:09.053 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.5 M 2024-05-09T03:01:09.053 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:09.090 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:09.217 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:09.217 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:09.217 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 127 k 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 127 k 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-05-09T03:01:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:09.416 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 408 kB/s | 77 kB 00:00 2024-05-09T03:01:09.417 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.0.0-3603.gc152e9f4.el9.x86_64 640 kB/s | 127 kB 00:00 2024-05-09T03:01:09.417 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:09.417 INFO:teuthology.orchestra.run.smithi181.stdout:Total 635 kB/s | 127 kB 00:00 2024-05-09T03:01:09.417 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:09.423 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:09.423 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:09.441 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-volume-19.0.0-3603.gc152e9f4.el9.no 1.2 MB/s | 273 kB 00:00 2024-05-09T03:01:09.442 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:09.442 INFO:teuthology.orchestra.run.smithi195.stdout:Total 898 kB/s | 350 kB 00:00 2024-05-09T03:01:09.455 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:09.465 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:09.465 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:09.488 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:09.488 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:09.499 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:09.499 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:09.640 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:09.644 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:09.748 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:09.842 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-09T03:01:09.885 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:09.902 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:09.902 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:09.902 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:01:09.903 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:10.021 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3603.gc152e9f4.e 377 kB/s | 142 kB 00:00 2024-05-09T03:01:10.232 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:10.338 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 357 kB/s | 248 kB 00:00 2024-05-09T03:01:10.447 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-05-09T03:01:10.485 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:10.485 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:10.485 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:10.485 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:10.485 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:10.486 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:10.506 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 869 kB/s | 51 kB 00:00 2024-05-09T03:01:10.531 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 299 kB/s | 58 kB 00:00 2024-05-09T03:01:10.651 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 1/2 2024-05-09T03:01:10.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-05-09T03:01:10.858 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-05-09T03:01:10.861 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:10.861 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.2 MB/s | 7.0 MB 00:05 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:10.927 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:10.935 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:10.947 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:10.947 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:11.045 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:11.045 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:11.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-05-09T03:01:11.350 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:11.357 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.5 MB/s | 6.0 MB 00:01 2024-05-09T03:01:11.357 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:11.357 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.0 MB/s | 7.0 MB 00:02 2024-05-09T03:01:11.392 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:45 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:11.431 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:11.442 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:11.443 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:11.475 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:11.516 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-09T03:01:11.518 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:11.520 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:11.520 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:11.541 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:11.541 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:11.568 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-09T03:01:11.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-05-09T03:01:11.809 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:48 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:11.847 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:11.932 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 127 k 2024-05-09T03:01:11.933 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 127 k 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 456 k 2024-05-09T03:01:11.934 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:12.023 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-09T03:01:12.085 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-09T03:01:12.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-09T03:01:12.167 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:46 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:12.187 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-09T03:01:12.251 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-09T03:01:12.272 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-19.0.0-3603.gc152e9f4.el9.x86_64 375 kB/s | 127 kB 00:00 2024-05-09T03:01:12.273 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:12.273 INFO:teuthology.orchestra.run.smithi195.stdout:Total 373 kB/s | 127 kB 00:00 2024-05-09T03:01:12.273 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:12.278 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:12.279 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:12.280 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:12.289 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:12.290 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:12.290 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 32 k 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-05-09T03:01:12.291 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:12.343 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:12.343 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:12.492 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:12.548 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.0.0-3603.gc152e9f4.el9.x86_6 124 kB/s | 32 kB 00:00 2024-05-09T03:01:12.548 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:12.548 INFO:teuthology.orchestra.run.smithi181.stdout:Total 123 kB/s | 32 kB 00:00 2024-05-09T03:01:12.549 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:12.552 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:12.552 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:12.574 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:12.574 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:12.588 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:12.609 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-09T03:01:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:12.699 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-09T03:01:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:12.772 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-09T03:01:12.802 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:13.011 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:13.034 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:13.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/6 2024-05-09T03:01:13.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-09T03:01:13.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-09T03:01:13.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-09T03:01:13.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-09T03:01:13.211 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:13.266 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:01:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:13.299 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:13.427 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:13.427 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:13.539 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-05-09T03:01:13.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-05-09T03:01:13.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-05-09T03:01:13.884 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 6/6 2024-05-09T03:01:13.885 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 1/6 2024-05-09T03:01:13.885 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-09T03:01:13.885 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-09T03:01:13.885 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-09T03:01:13.885 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-09T03:01:14.106 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:42 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:14.116 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:51 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:14.137 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:14.178 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:14.197 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:14.227 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 891 k 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-05-09T03:01:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:14.229 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 971 k 2024-05-09T03:01:14.229 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.8 M 2024-05-09T03:01:14.229 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:14.240 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:14.241 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:14.241 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:14.259 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:14.301 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:14.302 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:14.302 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:14.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-05-09T03:01:14.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-05-09T03:01:14.449 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-fuse 2024-05-09T03:01:14.527 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 452 kB/s | 80 kB 00:00 2024-05-09T03:01:14.668 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-fuse-19.0.0-3603.gc152e9f4.el9.x86_ 2.7 MB/s | 891 kB 00:00 2024-05-09T03:01:14.669 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:14.669 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.2 MB/s | 971 kB 00:00 2024-05-09T03:01:14.681 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:14.703 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:14.703 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:14.780 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:14.781 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:14.882 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:51 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:14.948 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:14.985 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:15.004 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 32 k 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:01:15.005 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:15.006 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 32 k 2024-05-09T03:01:15.006 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 142 k 2024-05-09T03:01:15.006 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:15.021 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:48 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:15.030 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:15.072 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:15.073 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:15.074 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-09T03:01:15.143 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 891 k 2024-05-09T03:01:15.144 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:15.145 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 971 k 2024-05-09T03:01:15.146 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 2.8 M 2024-05-09T03:01:15.146 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:15.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-05-09T03:01:15.178 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-19.0.0-3603.gc152e9f4.el9.x86_6 184 kB/s | 32 kB 00:00 2024-05-09T03:01:15.179 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:15.179 INFO:teuthology.orchestra.run.smithi195.stdout:Total 183 kB/s | 32 kB 00:00 2024-05-09T03:01:15.179 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:15.182 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:15.183 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:15.192 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:15.200 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:15.278 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:15.400 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:15.433 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 483 kB/s | 80 kB 00:00 2024-05-09T03:01:15.716 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:49 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:15.725 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-fuse-19.0.0-3603.gc152e9f4.el9.x86_ 1.9 MB/s | 891 kB 00:00 2024-05-09T03:01:15.725 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:15.726 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.6 MB/s | 971 kB 00:00 2024-05-09T03:01:15.737 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:15.758 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:15.758 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:15.797 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:15.803 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:15.803 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/2 2024-05-09T03:01:15.805 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:15.836 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:15.837 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:15.838 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:15.839 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:15.839 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:15.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-05-09T03:01:16.015 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:16.031 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:16.067 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:16.153 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-09T03:01:16.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-05-09T03:01:16.226 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:16.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-volume 2024-05-09T03:01:16.481 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:50 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:16.604 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:16.605 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:16.606 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:16.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-05-09T03:01:16.778 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:53 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:16.859 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:16.900 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:16.901 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:16.901 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:16.907 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:44 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:16.948 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2/2 2024-05-09T03:01:16.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/2 2024-05-09T03:01:16.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-05-09T03:01:17.032 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 273 k 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-09T03:01:17.033 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 350 k 2024-05-09T03:01:17.034 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.5 M 2024-05-09T03:01:17.035 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:17.212 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-09T03:01:17.212 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:17.213 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:17.213 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-09T03:01:17.213 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:17.213 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:17.268 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:51 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:17.349 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 467 kB/s | 77 kB 00:00 2024-05-09T03:01:17.360 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:17.403 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:17.404 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:17.404 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:17.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-05-09T03:01:17.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-volume 2024-05-09T03:01:17.542 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:54 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:17.557 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-volume-19.0.0-3603.gc152e9f4.el9.no 730 kB/s | 273 kB 00:00 2024-05-09T03:01:17.557 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:17.558 INFO:teuthology.orchestra.run.smithi096.stdout:Total 669 kB/s | 350 kB 00:00 2024-05-09T03:01:17.570 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:17.580 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:17.580 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:17.614 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:17.614 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:17.623 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:17.665 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:17.666 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:17.666 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:17.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-05-09T03:01:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:17.953 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-09T03:01:17.984 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:18.000 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:18.000 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:18.000 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:01:18.000 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:18.051 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:18.102 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:51 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:18.131 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:18.173 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:18.174 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-09T03:01:18.174 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:18.224 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:18.224 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:18.224 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-05-09T03:01:18.224 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:18.224 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-3603.gc152e9f4.el9 ceph-noarch 273 k 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:18.225 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 350 k 2024-05-09T03:01:18.226 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.5 M 2024-05-09T03:01:18.226 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:18.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-05-09T03:01:18.301 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:55 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:18.381 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:18.422 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:18.424 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:18.424 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:18.455 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 402 kB/s | 77 kB 00:00 2024-05-09T03:01:18.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-05-09T03:01:18.506 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-volume-19.0.0-3603.gc152e9f4.el9.no 1.1 MB/s | 273 kB 00:00 2024-05-09T03:01:18.507 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:18.507 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.2 MB/s | 350 kB 00:00 2024-05-09T03:01:18.519 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:18.529 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:18.529 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:18.563 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:18.564 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:18.679 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 1/2 2024-05-09T03:01:18.711 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:18.817 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:18.923 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-09T03:01:18.939 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:18.939 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:18.939 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 85 k 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:01:18.940 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:18.941 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-05-09T03:01:18.941 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-05-09T03:01:18.941 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:18.969 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:18.985 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2/2 2024-05-09T03:01:18.986 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:18.986 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:01:18.986 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:19.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:19.043 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:19.076 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:56 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:19.142 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 422 kB/s | 85 kB 00:00 2024-05-09T03:01:19.143 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:19.143 INFO:teuthology.orchestra.run.smithi181.stdout:Total 420 kB/s | 85 kB 00:00 2024-05-09T03:01:19.143 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:19.150 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:19.150 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:19.156 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:19.197 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:19.198 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:19.198 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:19.212 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:19.213 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:19.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-05-09T03:01:19.313 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-05-09T03:01:19.358 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:19.457 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:19.728 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 1/2 2024-05-09T03:01:19.821 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:19.842 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:56 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:19.893 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:47 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:19.922 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:19.964 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:19.964 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:19.965 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:20.000 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-3603.gc152e9f4.el9.noarch 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:20.001 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:20.017 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 127 k 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 127 k 2024-05-09T03:01:20.018 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-05-09T03:01:20.019 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:20.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:20.049 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:20.245 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-19.0.0-3603.gc152e9f4.el9.x86_64 559 kB/s | 127 kB 00:00 2024-05-09T03:01:20.246 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:20.246 INFO:teuthology.orchestra.run.smithi096.stdout:Total 555 kB/s | 127 kB 00:00 2024-05-09T03:01:20.246 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:20.252 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:20.252 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:20.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-05-09T03:01:20.315 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:20.316 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:20.383 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados-devel 2024-05-09T03:01:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:20.567 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:20.635 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:57 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:20.716 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:20.757 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:20.758 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-09T03:01:20.758 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:20.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-05-09T03:01:20.855 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:54 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:20.961 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:53 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.1 M 2024-05-09T03:01:20.978 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 13 M 2024-05-09T03:01:20.979 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:21.013 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:21.086 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 127 k 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:21.087 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 127 k 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 456 k 2024-05-09T03:01:21.088 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:21.298 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:21.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:21.299 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:21.299 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:21.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:21.299 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:21.339 INFO:teuthology.orchestra.run.smithi072.stdout:librados-devel-19.0.0-3603.gc152e9f4.el9.x86_64 505 kB/s | 127 kB 00:00 2024-05-09T03:01:21.340 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:21.340 INFO:teuthology.orchestra.run.smithi072.stdout:Total 502 kB/s | 127 kB 00:00 2024-05-09T03:01:21.340 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:21.345 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:21.345 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:21.404 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:58 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:21.409 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:21.409 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:21.454 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 6.5 MB/s | 3.1 MB 00:00 2024-05-09T03:01:21.455 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:21.456 INFO:teuthology.orchestra.run.smithi181.stdout:Total 6.5 MB/s | 3.1 MB 00:00 2024-05-09T03:01:21.456 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:21.465 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:21.465 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 85 k 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:21.527 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 85 k 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 231 k 2024-05-09T03:01:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:21.547 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:21.547 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:21.562 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:21.578 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-05-09T03:01:21.673 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:21.857 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 260 kB/s | 85 kB 00:00 2024-05-09T03:01:21.857 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:21.857 INFO:teuthology.orchestra.run.smithi195.stdout:Total 259 kB/s | 85 kB 00:00 2024-05-09T03:01:21.857 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:21.864 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:21.864 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:21.927 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:21.927 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:21.974 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:22.039 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:22.065 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:22.066 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:22.154 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.158 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:50 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:22.171 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.240 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:22.282 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:22.283 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:22.283 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:22.364 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-05-09T03:01:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:22.426 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:22.427 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:22.427 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:22.510 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.748 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs2 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:22.749 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:22.935 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:50 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:22.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-05-09T03:01:23.057 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 32 k 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:01:23.058 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:23.059 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 32 k 2024-05-09T03:01:23.059 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 142 k 2024-05-09T03:01:23.059 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:23.157 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:23.158 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:23.158 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:23.158 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:23.158 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:23.158 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:23.304 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-19.0.0-3603.gc152e9f4.el9.x86_6 129 kB/s | 32 kB 00:00 2024-05-09T03:01:23.305 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:23.305 INFO:teuthology.orchestra.run.smithi096.stdout:Total 128 kB/s | 32 kB 00:00 2024-05-09T03:01:23.305 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:23.309 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:23.309 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:23.314 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:56 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:23.323 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:23.324 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:23.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-05-09T03:01:23.396 INFO:teuthology.orchestra.run.smithi072.stdout:Package libcephfs2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:23.439 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:23.440 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:23.440 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:23.526 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:23.549 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs-devel 2024-05-09T03:01:23.568 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:00 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:23.691 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:23.692 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.1 M 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 13 M 2024-05-09T03:01:23.693 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:23.880 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:23.979 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:57 ago on Thu 09 May 2024 02:59:26 AM UTC. 2024-05-09T03:01:24.102 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-09T03:01:24.103 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:24.103 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:24.103 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:24.103 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 170 k 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 494 k 2024-05-09T03:01:24.104 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-09T03:01:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:24.106 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:24.107 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:24.137 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:57 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:24.259 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 32 k 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:24.260 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:01:24.261 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:24.261 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 32 k 2024-05-09T03:01:24.261 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 142 k 2024-05-09T03:01:24.261 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:24.277 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 5.3 MB/s | 3.1 MB 00:00 2024-05-09T03:01:24.278 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-05-09T03:01:24.279 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:24.279 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.3 MB/s | 3.1 MB 00:00 2024-05-09T03:01:24.279 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:24.287 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:24.287 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:24.341 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 719 kB/s | 170 kB 00:00 2024-05-09T03:01:24.342 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:24.342 INFO:teuthology.orchestra.run.smithi181.stdout:Total 715 kB/s | 170 kB 00:00 2024-05-09T03:01:24.342 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-09T03:01:24.350 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-09T03:01:24.350 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-09T03:01:24.366 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:24.367 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:24.413 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-09T03:01:24.414 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-09T03:01:24.510 INFO:teuthology.orchestra.run.smithi072.stdout:libcephfs-devel-19.0.0-3603.gc152e9f4.el9.x86_6 128 kB/s | 32 kB 00:00 2024-05-09T03:01:24.510 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:24.511 INFO:teuthology.orchestra.run.smithi072.stdout:Total 127 kB/s | 32 kB 00:00 2024-05-09T03:01:24.511 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:24.514 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:24.514 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:24.534 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:24.534 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:24.567 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-09T03:01:24.617 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:24.681 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:24.734 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:24.797 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:24.851 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:52 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:24.861 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:24.886 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:24.886 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:24.887 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:01:24.887 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:24.887 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:24.887 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:24.932 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:24.973 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:24.974 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:24.975 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:25.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-05-09T03:01:25.089 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:25.342 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:25.342 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:25.342 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:25.342 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:25.343 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:25.343 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:25.348 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-09T03:01:25.537 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados2 2024-05-09T03:01:25.569 DEBUG:teuthology.parallel:result is None 2024-05-09T03:01:25.625 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:53 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:25.711 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:25.754 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:25.755 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:25.755 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:25.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-05-09T03:01:25.964 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:25.964 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:25.964 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:25.965 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:25.965 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:25.965 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:26.110 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:59 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:26.190 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:26.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-05-09T03:01:26.231 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:26.232 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:26.232 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:26.315 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librbd1 2024-05-09T03:01:26.428 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:54 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:26.510 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:26.553 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:26.554 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:26.554 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:26.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-05-09T03:01:26.795 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:03 ago on Thu 09 May 2024 02:59:23 AM UTC. 2024-05-09T03:01:26.882 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:59 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:26.919 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 170 k 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-09T03:01:26.920 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:26.921 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 170 k 2024-05-09T03:01:26.921 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 494 k 2024-05-09T03:01:26.921 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-09T03:01:26.963 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:27.004 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:27.005 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:27.005 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:27.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rados 2024-05-09T03:01:27.191 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:55 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:27.264 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 497 kB/s | 170 kB 00:00 2024-05-09T03:01:27.265 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:27.265 INFO:teuthology.orchestra.run.smithi195.stdout:Total 495 kB/s | 170 kB 00:00 2024-05-09T03:01:27.265 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-09T03:01:27.270 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:27.273 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-09T03:01:27.273 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-09T03:01:27.311 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:27.312 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:27.312 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:27.336 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-09T03:01:27.336 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-09T03:01:27.382 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-05-09T03:01:27.500 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-09T03:01:27.604 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:27.656 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:00 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:27.737 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rados-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:27.778 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:27.779 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:27.779 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:27.845 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rgw 2024-05-09T03:01:27.948 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:55 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:28.030 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:28.069 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:28.070 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:28.071 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:28.144 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:28.264 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-09T03:01:28.412 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:01 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:28.492 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rgw-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:28.534 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:28.534 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:28.535 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:28.538 DEBUG:teuthology.parallel:result is None 2024-05-09T03:01:28.611 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-cephfs 2024-05-09T03:01:28.708 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:56 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:28.787 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:28.829 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:28.830 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-09T03:01:28.830 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:28.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-05-09T03:01:29.175 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:02 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:29.257 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-cephfs-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:29.297 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:29.298 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:29.298 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:29.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rbd 2024-05-09T03:01:29.478 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:57 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:29.599 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 85 k 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:01:29.600 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:29.601 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 85 k 2024-05-09T03:01:29.601 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 231 k 2024-05-09T03:01:29.601 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:29.808 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 413 kB/s | 85 kB 00:00 2024-05-09T03:01:29.808 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:29.808 INFO:teuthology.orchestra.run.smithi096.stdout:Total 410 kB/s | 85 kB 00:00 2024-05-09T03:01:29.809 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:29.816 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:29.816 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:29.879 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:29.879 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:29.942 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:02 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:30.022 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 is already installed. 2024-05-09T03:01:30.037 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:30.063 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:30.064 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-05-09T03:01:30.064 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:30.131 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:30.144 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-fuse 2024-05-09T03:01:30.486 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:30.711 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:03 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:30.761 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:30.762 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:30.832 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:30.833 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:30.833 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:30.833 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:30.833 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:30.833 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 85 k 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 85 k 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 231 k 2024-05-09T03:01:30.834 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:31.068 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-05-09T03:01:31.161 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-fuse-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 262 kB/s | 85 kB 00:00 2024-05-09T03:01:31.161 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:31.161 INFO:teuthology.orchestra.run.smithi072.stdout:Total 261 kB/s | 85 kB 00:00 2024-05-09T03:01:31.161 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:31.168 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:31.168 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:31.371 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:31.474 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:31.638 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:59 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:31.760 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:01:31.761 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:31.762 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.1 M 2024-05-09T03:01:31.762 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 13 M 2024-05-09T03:01:31.762 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:31.838 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:32.078 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:32.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:32.078 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:32.079 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:32.079 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:32.079 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:32.145 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-05-09T03:01:32.146 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:32.146 INFO:teuthology.orchestra.run.smithi096.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-05-09T03:01:32.146 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:32.155 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:32.155 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:32.236 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:32.236 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:32.367 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-mirror 2024-05-09T03:01:32.662 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:32.747 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:32.772 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:32.937 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:05 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:33.061 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 3.1 M 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:01:33.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:33.063 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.1 M 2024-05-09T03:01:33.063 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 13 M 2024-05-09T03:01:33.063 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:33.453 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-mirror-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-05-09T03:01:33.454 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:33.455 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-05-09T03:01:33.455 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:33.463 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:33.464 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:33.543 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:33.543 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:33.788 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:33.788 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:33.789 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:33.789 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:33.789 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:33.789 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:33.968 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:34.030 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:34.048 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:01:34.055 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:34.627 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:02 ago on Thu 09 May 2024 02:59:32 AM UTC. 2024-05-09T03:01:34.754 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-09T03:01:34.754 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 170 k 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 170 k 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 494 k 2024-05-09T03:01:34.755 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-09T03:01:34.983 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 750 kB/s | 170 kB 00:00 2024-05-09T03:01:34.984 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:34.984 INFO:teuthology.orchestra.run.smithi096.stdout:Total 745 kB/s | 170 kB 00:00 2024-05-09T03:01:34.984 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-09T03:01:34.991 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-09T03:01:34.992 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-09T03:01:35.055 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-09T03:01:35.055 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-09T03:01:35.203 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-09T03:01:35.306 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:35.344 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:35.620 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-nbd 2024-05-09T03:01:35.744 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:35.967 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-09T03:01:36.202 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:09 ago on Thu 09 May 2024 02:59:27 AM UTC. 2024-05-09T03:01:36.239 DEBUG:teuthology.parallel:result is None 2024-05-09T03:01:36.325 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-3603.gc152e9f4.el9 ceph 170 k 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-05-09T03:01:36.326 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:36.327 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 170 k 2024-05-09T03:01:36.327 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 494 k 2024-05-09T03:01:36.327 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-05-09T03:01:36.557 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-nbd-19.0.0-3603.gc152e9f4.el9.x86_64.rpm 739 kB/s | 170 kB 00:00 2024-05-09T03:01:36.557 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:01:36.558 INFO:teuthology.orchestra.run.smithi072.stdout:Total 735 kB/s | 170 kB 00:00 2024-05-09T03:01:36.558 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-05-09T03:01:36.565 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-05-09T03:01:36.565 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-05-09T03:01:36.630 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-05-09T03:01:36.630 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-05-09T03:01:36.782 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-05-09T03:01:36.889 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:37.344 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:37.599 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 1/1 2024-05-09T03:01:37.599 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:37.599 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-05-09T03:01:37.600 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-3603.gc152e9f4.el9.x86_64 2024-05-09T03:01:37.600 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:37.600 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-05-09T03:01:37.908 DEBUG:teuthology.parallel:result is None 2024-05-09T03:01:37.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T03:01:38.062 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:01:38.083 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.083 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.084 INFO:teuthology.task.install:The correct ceph version 19.0.0-3603.gc152e9f4 is installed. 2024-05-09T03:01:38.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T03:01:38.303 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:01:38.329 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.329 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.329 INFO:teuthology.task.install:The correct ceph version 19.0.0-3603.gc152e9f4 is installed. 2024-05-09T03:01:38.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T03:01:38.548 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:01:38.573 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.574 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.574 INFO:teuthology.task.install:The correct ceph version 19.0.0-3603.gc152e9f4 is installed. 2024-05-09T03:01:38.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c152e9f40d11b517c5526b6e8144d52000d7e690 2024-05-09T03:01:38.720 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:01:38.746 INFO:teuthology.orchestra.run.smithi195.stdout:19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.746 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3603.gc152e9f4.el9 2024-05-09T03:01:38.746 INFO:teuthology.task.install:The correct ceph version 19.0.0-3603.gc152e9f4 is installed. 2024-05-09T03:01:38.748 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T03:01:38.749 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:38.749 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:01:38.775 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:38.775 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:01:38.806 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:38.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:01:38.836 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:38.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:01:38.867 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T03:01:38.867 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:38.867 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:01:38.892 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:01:38.957 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:38.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:01:38.987 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:01:39.055 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:39.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:01:39.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:01:39.153 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:39.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:01:39.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:01:39.252 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T03:01:39.252 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:39.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:01:39.277 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:01:39.341 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:39.341 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:01:39.371 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:01:39.440 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:39.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:01:39.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:01:39.531 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:39.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:01:39.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:01:39.633 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T03:01:39.633 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:39.633 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:01:39.658 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:01:39.722 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:39.722 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:01:39.751 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:01:39.821 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:39.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:01:39.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:01:39.910 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:39.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:01:39.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:01:40.000 INFO:teuthology.run_tasks:Running task ceph... 2024-05-09T03:01:40.086 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-09T03:01:40.087 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2024-05-09T03:01:40.089 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-05-09T03:01:40.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-05-09T03:01:40.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2024-05-09T03:01:40.118 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-09T03:01:40.118 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:01:40.155 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:01:40.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:01:40.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:01:40.188 INFO:tasks.ceph:Creating extra log directories... 2024-05-09T03:01:40.188 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:01:40.220 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:01:40.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:01:40.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:01:40.270 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-09T03:01:40.270 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-09T03:01:40.270 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'branch': 'wip-sjust-balanced-read-testing-2024-05-08', 'description': 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced}', 'email': None, 'first_in_suite': False, 'job_id': '7699454', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}, {'volumes': {'size': 50}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sjust-balanced-read-testing-2024-05-08'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-sjust-balanced-read-testing-2024-05-08', 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690'}}, 'owner': 'scheduled_sjust@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 4494, 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'sleep_before_teardown': 0, 'subset': '10/3000', 'suite': 'rados:thrash', 'suite_branch': 'wip-sjust-balanced-read-testing-2024-05-08', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'targets': {'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAz2REg4lMucipWe/Qym4OExMaGQThHxYzzcoO5SnYyhhtPFTUBV1VOFUK4Im5i5selAYjBOh5T04h7agwIORdE=', 'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEEG9lnjv2iuYW/yBzKLhScFVmU519RnxmwnqXsoMDNaGITzCPfZU3OoyDf6nQ2Q4iPsTD2D9CbR6lllHxsPs0=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWZvl6DWaFvDjAubG4RwJqIdDPPiqZ7+JXnpDETpJZ+YnyXfJCg9qp3bhHt2WlZCt7MkzgHzvZKEwQWtRqRi5A=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY4lZiaVWRxSjv4ZaODKGlHFKK4EnLuUQ37HMQtVXu99F/L6Kjnnbd5S8OdOHhcGHfZDVr05ctdvD6JJ63RnYw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c152e9f40d11b517c5526b6e8144d52000d7e690', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['ceph config set mgr mgr/balancer/log_level debug', 'ceph balancer status', 'ceph osd set-require-min-compat-client reef', 'ceph balancer mode upmap-read', 'ceph balancer on', 'ceph balancer status', 'ceph balancer status detail']}}, {'thrashosds': {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'min_in': 2, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 3600', 'sudo ceph osd pool set cache target_max_objects 250', 'sudo ceph osd pool set cache min_read_recency_for_promote 2']}}, {'rados': {'balance_reads': True, 'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-09_02:35:04', 'tube': 'smithi', 'user': 'sjust', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-09T03:01:40.271 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:01:40.285 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:01:40.300 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:01:40.302 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:01:40.318 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:01:40.340 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:01:40.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:01:40.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:01:40.386 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:40.386 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:01:40.419 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:01:40.419 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-05-09 03:01:34.316081415 +0000 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-05-09 02:58:43.286568978 +0000 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-05-09 02:58:43.286568978 +0000 2024-05-09T03:01:40.474 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-05-09 02:58:43.286568978 +0000 2024-05-09T03:01:40.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:01:40.539 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-05-09T03:01:40.540 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-05-09T03:01:40.540 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.00023939 s, 2.1 MB/s 2024-05-09T03:01:40.541 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:01:40.597 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-05-09 03:01:34.316081415 +0000 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-05-09 02:58:43.633561878 +0000 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-05-09 02:58:43.633561878 +0000 2024-05-09T03:01:40.653 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-05-09 02:58:43.633561878 +0000 2024-05-09T03:01:40.654 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:01:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-05-09T03:01:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-05-09T03:01:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000222174 s, 2.3 MB/s 2024-05-09T03:01:40.717 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:01:40.773 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-05-09 03:01:34.316081415 +0000 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-05-09 02:58:43.985554676 +0000 2024-05-09T03:01:40.828 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-05-09 02:58:43.985554676 +0000 2024-05-09T03:01:40.829 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-05-09 02:58:43.985554676 +0000 2024-05-09T03:01:40.829 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:01:40.891 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-05-09T03:01:40.891 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-05-09T03:01:40.891 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000226347 s, 2.3 MB/s 2024-05-09T03:01:40.892 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:01:40.948 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-05-09T03:01:41.004 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-05-09 03:01:34.317081395 +0000 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-05-09 02:58:44.317547883 +0000 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-05-09 02:58:44.317547883 +0000 2024-05-09T03:01:41.005 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-05-09 02:58:44.317547883 +0000 2024-05-09T03:01:41.005 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:01:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-05-09T03:01:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-05-09T03:01:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000197021 s, 2.6 MB/s 2024-05-09T03:01:41.069 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:01:41.126 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:41.126 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:41.127 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:01:41.146 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:01:41.146 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-05-09T03:01:41.204 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-09 03:01:33.030677456 +0000 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-09 02:58:40.772526582 +0000 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-09 02:58:40.772526582 +0000 2024-05-09T03:01:41.205 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-09 02:58:40.772526582 +0000 2024-05-09T03:01:41.205 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:01:41.271 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-09T03:01:41.272 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-09T03:01:41.272 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000222602 s, 2.3 MB/s 2024-05-09T03:01:41.273 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:01:41.332 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-05-09T03:01:41.389 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:01:41.389 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.389 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-09T03:01:41.389 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.389 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.390 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-09 03:01:33.031677433 +0000 2024-05-09T03:01:41.390 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-09 02:58:41.099519097 +0000 2024-05-09T03:01:41.390 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-09 02:58:41.099519097 +0000 2024-05-09T03:01:41.390 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-09 02:58:41.099519097 +0000 2024-05-09T03:01:41.390 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:01:41.455 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-09T03:01:41.455 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-09T03:01:41.455 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000218151 s, 2.3 MB/s 2024-05-09T03:01:41.457 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:01:41.514 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-09 03:01:33.031677433 +0000 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-09 02:58:41.466510697 +0000 2024-05-09T03:01:41.571 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-09 02:58:41.466510697 +0000 2024-05-09T03:01:41.572 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-09 02:58:41.466510697 +0000 2024-05-09T03:01:41.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:01:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-09T03:01:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-09T03:01:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000200211 s, 2.6 MB/s 2024-05-09T03:01:41.637 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:01:41.694 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-05-09T03:01:41.750 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:01:41.750 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-09 03:01:33.032677411 +0000 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-09 02:58:41.798503098 +0000 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-09 02:58:41.798503098 +0000 2024-05-09T03:01:41.754 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-09 02:58:41.798503098 +0000 2024-05-09T03:01:41.754 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:01:41.814 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-09T03:01:41.814 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-09T03:01:41.814 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000223342 s, 2.3 MB/s 2024-05-09T03:01:41.815 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:01:41.872 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:41.872 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:41.873 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:01:41.892 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:01:41.892 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:41.951 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-09 03:01:22.294469021 +0000 2024-05-09T03:01:41.952 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-09 02:58:37.455398265 +0000 2024-05-09T03:01:41.952 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-09 02:58:37.455398265 +0000 2024-05-09T03:01:41.952 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-09 02:58:37.455398265 +0000 2024-05-09T03:01:41.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:01:42.019 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-09T03:01:42.019 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-09T03:01:42.019 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000216339 s, 2.4 MB/s 2024-05-09T03:01:42.020 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:01:42.080 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-05-09T03:01:42.138 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:01:42.138 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:42.138 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-09 03:01:22.294469021 +0000 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-09 02:58:37.802391629 +0000 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-09 02:58:37.802391629 +0000 2024-05-09T03:01:42.139 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-09 02:58:37.802391629 +0000 2024-05-09T03:01:42.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:01:42.204 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-09T03:01:42.205 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-09T03:01:42.205 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000227481 s, 2.3 MB/s 2024-05-09T03:01:42.206 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:01:42.263 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-05-09T03:01:42.320 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-09 03:01:22.295469001 +0000 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-09 02:58:38.142385127 +0000 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-09 02:58:38.142385127 +0000 2024-05-09T03:01:42.321 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-09 02:58:38.142385127 +0000 2024-05-09T03:01:42.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:01:42.385 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-09T03:01:42.385 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-09T03:01:42.385 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000216027 s, 2.4 MB/s 2024-05-09T03:01:42.386 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:01:42.444 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-05-09T03:01:42.499 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-09 03:01:22.295469001 +0000 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-09 02:58:38.468378893 +0000 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-09 02:58:38.468378893 +0000 2024-05-09T03:01:42.500 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-09 02:58:38.468378893 +0000 2024-05-09T03:01:42.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:01:42.563 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-09T03:01:42.564 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-09T03:01:42.564 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000218754 s, 2.3 MB/s 2024-05-09T03:01:42.565 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:01:42.622 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:42.622 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:42.622 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:01:42.643 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:01:42.643 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-05-09T03:01:42.701 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:01:42.701 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:42.701 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-09T03:01:42.701 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:42.701 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:42.702 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-09 03:01:25.138133602 +0000 2024-05-09T03:01:42.702 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-09 02:58:41.894363169 +0000 2024-05-09T03:01:42.702 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-09 02:58:41.894363169 +0000 2024-05-09T03:01:42.702 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-09 02:58:41.894363169 +0000 2024-05-09T03:01:42.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:01:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-09T03:01:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-09T03:01:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000114878 s, 4.5 MB/s 2024-05-09T03:01:42.769 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:01:42.827 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-05-09T03:01:42.884 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:01:42.884 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:42.884 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-09T03:01:42.884 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:42.885 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:42.885 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-09 03:01:25.138133602 +0000 2024-05-09T03:01:42.885 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-09 02:58:42.231357101 +0000 2024-05-09T03:01:42.885 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-09 02:58:42.231357101 +0000 2024-05-09T03:01:42.885 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-09 02:58:42.231357101 +0000 2024-05-09T03:01:42.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:01:42.950 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-09T03:01:42.950 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-09T03:01:42.950 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000108609 s, 4.7 MB/s 2024-05-09T03:01:42.951 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:01:43.009 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-05-09T03:01:43.065 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:01:43.065 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:43.065 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-09 03:01:25.139133584 +0000 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-09 02:58:42.554351285 +0000 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-09 02:58:42.554351285 +0000 2024-05-09T03:01:43.066 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-09 02:58:42.554351285 +0000 2024-05-09T03:01:43.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:01:43.129 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-09T03:01:43.129 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-09T03:01:43.130 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000107804 s, 4.7 MB/s 2024-05-09T03:01:43.131 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:01:43.190 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-05-09T03:01:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:01:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:01:43.246 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-09T03:01:43.246 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:01:43.246 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:01:43.247 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-09 03:01:25.139133584 +0000 2024-05-09T03:01:43.247 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-09 02:58:42.898345090 +0000 2024-05-09T03:01:43.247 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-09 02:58:42.898345090 +0000 2024-05-09T03:01:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-09 02:58:42.898345090 +0000 2024-05-09T03:01:43.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:01:43.310 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-09T03:01:43.310 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-09T03:01:43.310 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000104238 s, 4.9 MB/s 2024-05-09T03:01:43.312 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:01:43.369 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:43.369 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-05-09T03:01:43.369 INFO:tasks.ceph:Generating config... 2024-05-09T03:01:43.370 INFO:tasks.ceph:[osd] debug monc = 20 2024-05-09T03:01:43.370 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-09T03:01:43.370 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-09T03:01:43.370 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-09T03:01:43.370 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] filestore queue throttle high multiple = 2 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] filestore queue throttle max multiple = 10 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] journal throttle high multiple = 2 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] journal throttle max multiple = 10 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-09T03:01:43.371 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd max backfills = 9 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd max scrubs = 1 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-09T03:01:43.372 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-09T03:01:43.373 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-05-09T03:01:43.373 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-05-09T03:01:43.373 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-09T03:01:43.373 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2024-05-09T03:01:43.373 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] ms type = async 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-05-09T03:01:43.373 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-05-09T03:01:43.373 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-09T03:01:43.373 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-09T03:01:43.374 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-09T03:01:43.374 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-09T03:01:43.374 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-09T03:01:43.374 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-09T03:01:43.374 INFO:tasks.ceph:Setting up mon.a... 2024-05-09T03:01:43.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-09T03:01:43.450 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.keyring 2024-05-09T03:01:43.454 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-09T03:01:43.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:01:43.525 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.72'), ('mon.b', '172.21.15.96'), ('mon.c', '172.21.15.181')] 2024-05-09T03:01:43.526 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.72,172.21.15.96,172.21.15.181', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-05-09T03:01:43.527 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:43.527 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T03:01:43.582 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.72 --add b 172.21.15.96 --add c 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:43.668 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:43.668 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: generated fsid f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.668 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-05-09T03:01:43.668 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 0 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:fsid f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:last_changed 2024-05-09T03:01:43.668369+0000 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-05-09T03:01:43.668369+0000 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:min_mon_release 17 (quincy) 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:election_strategy: 1 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:1: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:2: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2024-05-09T03:01:43.669 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-09T03:01:43.671 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T03:01:43.686 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f061f132-a796-4bcc-9d93-8f60c6bd3094... 2024-05-09T03:01:43.687 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:01:43.728 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:01:43.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:01:43.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:01:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-05-09T03:01:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-05-09T03:01:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:01:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:01:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.96,172.21.15.181" 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon client directed command retry = 5 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: mon election default strategy = 1 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: ms bind msgr1 = False 2024-05-09T03:01:43.775 INFO:teuthology.orchestra.run.smithi072.stdout: ms bind msgr2 = True 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: ms inject socket failures = 5000 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: ms type = async 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_min_size = 2 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_size = 2 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 120 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.776 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.777 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 20 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore zero block detection = True 2024-05-09T03:01:43.778 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: filestore queue throttle high multiple = 2 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: filestore queue throttle max multiple = 10 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: journal throttle high multiple = 2 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: journal throttle max multiple = 10 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: osd backoff on peering = True 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: osd blocked scrub grace period = 3600 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify cached snaps = True 2024-05-09T03:01:43.779 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify missing on start = True 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd heartbeat use min delay socket = True 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd inject bad map crc probability = 0.1 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd max backfills = 9 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd max markdown count = 1000 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd max scrubs = 1 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock override recovery settings = True 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd op queue = debug_random 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd op queue cut off = debug_random 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub min interval = 60 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd shutdown pgref assert = True 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd_max_pg_log_entries = 3000 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: osd_min_pg_log_entries = 3000 2024-05-09T03:01:43.780 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-05-09T03:01:43.781 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-05-09T03:01:43.782 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: mon scrub interval = 300 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-05-09T03:01:43.783 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-05-09T03:01:43.784 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-05-09T03:01:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-05-09T03:01:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:01:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:01:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.72,172.21.15.96,172.21.15.181" 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: mon election default strategy = 1 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: ms bind msgr1 = False 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: ms bind msgr2 = True 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: ms type = async 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_min_size = 2 2024-05-09T03:01:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_size = 2 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 120 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-05-09T03:01:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bdev async discard = True 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bdev enable discard = True 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore allocator = bitmap 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore zero block detection = True 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: filestore queue throttle high multiple = 2 2024-05-09T03:01:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: filestore queue throttle max multiple = 10 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: journal throttle high multiple = 2 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: journal throttle max multiple = 10 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd backoff on peering = True 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd blocked scrub grace period = 3600 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify cached snaps = True 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify missing on start = True 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd heartbeat use min delay socket = True 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd inject bad map crc probability = 0.1 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd max backfills = 9 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd max scrubs = 1 2024-05-09T03:01:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock override recovery settings = True 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue = debug_random 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue cut off = debug_random 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub min interval = 60 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd shutdown pgref assert = True 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd_max_pg_log_entries = 3000 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: osd_min_pg_log_entries = 3000 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-09T03:01:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: mon scrub interval = 300 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-09T03:01:43.794 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-05-09T03:01:43.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.796 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.797 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.72,172.21.15.96,172.21.15.181" 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: mon client directed command retry = 5 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 1 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: ms bind msgr1 = False 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: ms bind msgr2 = True 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject socket failures = 5000 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: ms type = async 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_min_size = 2 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_size = 2 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.798 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 120 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-05-09T03:01:43.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-05-09T03:01:43.800 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: filestore queue throttle high multiple = 2 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: filestore queue throttle max multiple = 10 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: journal throttle high multiple = 2 2024-05-09T03:01:43.801 INFO:teuthology.orchestra.run.smithi181.stdout: journal throttle max multiple = 10 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd backoff on peering = True 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd blocked scrub grace period = 3600 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd heartbeat use min delay socket = True 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd inject bad map crc probability = 0.1 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd max backfills = 9 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd max markdown count = 1000 2024-05-09T03:01:43.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd max scrubs = 1 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub min interval = 60 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd_max_pg_log_entries = 3000 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd_min_pg_log_entries = 3000 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-05-09T03:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.804 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-05-09T03:01:43.805 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-05-09T03:01:43.806 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:01:43.807 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:01:43.808 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "172.21.15.72,172.21.15.96,172.21.15.181" 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: mon election default strategy = 1 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: ms bind msgr1 = False 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: ms bind msgr2 = True 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: ms type = async 2024-05-09T03:01:43.809 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_min_size = 2 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_size = 2 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = f061f132-a796-4bcc-9d93-8f60c6bd3094 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 120 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.810 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:01:43.811 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug monc = 20 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bdev async discard = True 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bdev enable discard = True 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore allocator = bitmap 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore zero block detection = True 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 20 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 20 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-05-09T03:01:43.812 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 10 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: filestore queue throttle high multiple = 2 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: filestore queue throttle max multiple = 10 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: journal throttle high multiple = 2 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: journal throttle max multiple = 10 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd backoff on peering = True 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd blocked scrub grace period = 3600 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify cached snaps = True 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify missing on start = True 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd heartbeat use min delay socket = True 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd inject bad map crc probability = 0.1 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd max backfills = 9 2024-05-09T03:01:43.813 INFO:teuthology.orchestra.run.smithi096.stdout: osd max markdown count = 1000 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd max scrubs = 1 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock override recovery settings = True 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd op queue = debug_random 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd op queue cut off = debug_random 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub min interval = 60 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd shutdown pgref assert = True 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd_max_pg_log_entries = 3000 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: osd_min_pg_log_entries = 3000 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-05-09T03:01:43.814 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-05-09T03:01:43.815 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: mon scrub interval = 300 2024-05-09T03:01:43.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2024-05-09T03:01:43.817 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2024-05-09T03:01:43.820 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-09T03:01:43.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-09T03:01:43.913 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-09T03:01:43.913 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:43.914 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-09T03:01:43.929 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:43.929 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-09T03:01:43.984 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:01:43.984 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:43.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:01:43.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:01:44.061 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:01:44.061 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:44.116 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:01:44.116 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:44.117 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:01:44.117 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:01:44.157 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:01:44.157 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:44.216 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:01:44.216 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:44.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:01:44.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:01:44.256 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:01:44.256 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:44.314 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:01:44.314 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:44.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:01:44.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:01:44.354 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:01:44.355 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:44.411 INFO:tasks.ceph:Setting up mon nodes... 2024-05-09T03:01:44.412 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-09T03:01:44.412 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T03:01:44.463 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T03:01:44.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T03:01:44.536 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T03:01:44.539 INFO:tasks.ceph:Setting up mds nodes... 2024-05-09T03:01:44.539 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-09T03:01:44.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T03:01:44.582 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-09T03:01:44.595 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-09T03:01:44.595 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-05-09T03:01:44.595 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-09T03:01:44.619 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:44.619 INFO:tasks.ceph:role: osd.0 2024-05-09T03:01:44.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:01:44.620 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:01:44.691 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:01:44.692 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:01:44.692 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:01:44.692 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:01:45.196 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-05-09T03:01:45.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-05-09T03:01:45.198 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-09T03:01:45.343 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-09T03:01:45.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-09T03:01:45.441 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:45.441 INFO:tasks.ceph:role: osd.4 2024-05-09T03:01:45.441 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:01:45.442 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:01:45.513 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:01:45.513 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:01:45.514 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:01:45.960 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-05-09T03:01:45.962 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-05-09T03:01:45.963 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-05-09T03:01:46.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-09T03:01:46.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-05-09T03:01:46.154 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:46.154 INFO:tasks.ceph:role: osd.8 2024-05-09T03:01:46.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:01:46.154 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:01:46.225 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:01:46.226 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:01:46.711 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-05-09T03:01:46.713 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-05-09T03:01:46.713 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-05-09T03:01:46.863 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-05-09T03:01:46.893 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-05-09T03:01:46.958 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:46.959 INFO:tasks.ceph:role: osd.12 2024-05-09T03:01:46.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:01:46.959 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:01:47.031 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:01:47.552 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-05-09T03:01:47.554 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-05-09T03:01:47.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-05-09T03:01:47.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-05-09T03:01:47.748 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.882+0000 7fad6240c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.883+0000 7fad6240c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.883+0000 7fad6240c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.883+0000 7fad6240c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.883+0000 7fad6240c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:01:47.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:47.883+0000 7fad6240c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-09T03:01:50.730 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-09T03:01:50.758 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.846+0000 7f824101c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.846+0000 7f824101c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.846+0000 7f824101c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.846+0000 7f824101c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.846+0000 7f824101c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:01:50.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:50.847+0000 7f824101c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-09T03:01:53.699 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-09T03:01:53.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.812+0000 7f13ef40d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.813+0000 7f13ef40d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.813+0000 7f13ef40d640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.813+0000 7f13ef40d640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.813+0000 7f13ef40d640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-09T03:01:53.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:53.813+0000 7f13ef40d640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-05-09T03:01:56.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-05-09T03:01:56.691 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.777+0000 7ff8a92a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.777+0000 7ff8a92a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.777+0000 7ff8a92a2640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.777+0000 7ff8a92a2640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.777+0000 7ff8a92a2640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-05-09T03:01:56.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-05-09T03:01:56.778+0000 7ff8a92a2640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-05-09T03:01:59.628 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-05-09T03:01:59.656 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-09T03:01:59.686 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-05-09T03:01:59.686 INFO:tasks.ceph:role: osd.1 2024-05-09T03:01:59.686 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:01:59.686 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:01:59.762 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:01:59.763 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:01:59.763 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:01:59.763 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:01:59.763 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:00.224 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-09T03:02:00.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-09T03:02:00.226 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-05-09T03:02:00.357 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-09T03:02:00.389 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-09T03:02:00.454 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:00.455 INFO:tasks.ceph:role: osd.5 2024-05-09T03:02:00.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:02:00.455 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:00.527 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:00.528 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:00.528 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:00.980 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-09T03:02:00.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-09T03:02:00.983 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-09T03:02:01.053 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-09T03:02:01.123 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-05-09T03:02:01.189 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:01.189 INFO:tasks.ceph:role: osd.9 2024-05-09T03:02:01.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:02:01.189 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:02:01.261 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:01.262 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:01.751 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-09T03:02:01.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-09T03:02:01.753 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-05-09T03:02:01.859 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-05-09T03:02:01.889 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-05-09T03:02:01.955 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:01.956 INFO:tasks.ceph:role: osd.13 2024-05-09T03:02:01.956 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:02:01.956 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:02.029 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:02.030 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:02.030 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:02.517 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-09T03:02:02.519 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-09T03:02:02.520 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-05-09T03:02:02.658 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-05-09T03:02:02.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:02.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.784+0000 7fe9a0c15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-09T03:02:02.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.785+0000 7fe9a0c15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-09T03:02:02.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.785+0000 7fe9a0c15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:02:02.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.785+0000 7fe9a0c15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:02:02.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.785+0000 7fe9a0c15640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:02:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:02.790+0000 7fe9a0c15640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-09T03:02:05.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-09T03:02:05.679 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.772+0000 7f633041e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.772+0000 7f633041e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.772+0000 7f633041e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.772+0000 7f633041e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.772+0000 7f633041e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:02:05.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:05.773+0000 7f633041e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-09T03:02:08.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-09T03:02:08.650 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:08.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.739+0000 7f409be19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-05-09T03:02:08.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.739+0000 7f409be19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-05-09T03:02:08.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.740+0000 7f409be19640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-09T03:02:08.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.740+0000 7f409be19640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-09T03:02:08.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.740+0000 7f409be19640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-09T03:02:08.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:08.740+0000 7f409be19640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-05-09T03:02:11.596 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-05-09T03:02:11.623 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-05-09T03:02:11.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-09T03:02:11.711+0000 7faa38799640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-05-09T03:02:14.559 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-05-09T03:02:14.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-09T03:02:14.618 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:14.618 INFO:tasks.ceph:role: osd.2 2024-05-09T03:02:14.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:02:14.619 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:14.695 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:15.193 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-05-09T03:02:15.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-05-09T03:02:15.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-05-09T03:02:15.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-09T03:02:15.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-09T03:02:15.427 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:15.428 INFO:tasks.ceph:role: osd.6 2024-05-09T03:02:15.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:02:15.428 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:15.501 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:15.502 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:15.502 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:15.502 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:15.502 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:15.957 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-05-09T03:02:15.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-05-09T03:02:15.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-05-09T03:02:16.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-09T03:02:16.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-05-09T03:02:16.143 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:16.143 INFO:tasks.ceph:role: osd.10 2024-05-09T03:02:16.143 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:02:16.144 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:02:16.215 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:16.216 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-05-09T03:02:16.705 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-05-09T03:02:16.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-05-09T03:02:16.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-05-09T03:02:16.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-05-09T03:02:16.910 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:16.910 INFO:tasks.ceph:role: osd.14 2024-05-09T03:02:16.910 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:02:16.910 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:16.982 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:16.983 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:16.983 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:16.983 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:16.983 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:17.491 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-05-09T03:02:17.493 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-05-09T03:02:17.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-05-09T03:02:17.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-05-09T03:02:17.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.757+0000 7f82e56d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.758+0000 7f82e56d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.758+0000 7f82e56d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.758+0000 7f82e56d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.758+0000 7f82e56d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:02:17.759 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:17.758+0000 7f82e56d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-09T03:02:20.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-09T03:02:20.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:20.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.722+0000 7f2cf9212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-09T03:02:20.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.722+0000 7f2cf9212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-09T03:02:20.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.723+0000 7f2cf9212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:02:20.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.723+0000 7f2cf9212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:02:20.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.723+0000 7f2cf9212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:02:20.724 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:20.723+0000 7f2cf9212640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-09T03:02:23.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-09T03:02:23.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.687+0000 7ffbde023640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.688+0000 7ffbde023640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.688+0000 7ffbde023640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.688+0000 7ffbde023640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.688+0000 7ffbde023640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-05-09T03:02:23.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:23.688+0000 7ffbde023640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-05-09T03:02:26.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-05-09T03:02:26.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:26.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.652+0000 7f0defa21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-05-09T03:02:26.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.653+0000 7f0defa21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-05-09T03:02:26.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.653+0000 7f0defa21640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-05-09T03:02:26.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.653+0000 7f0defa21640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-05-09T03:02:26.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.653+0000 7f0defa21640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-05-09T03:02:26.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-05-09T03:02:26.653+0000 7f0defa21640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-05-09T03:02:29.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-05-09T03:02:29.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-09T03:02:29.571 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:29.571 INFO:tasks.ceph:role: osd.3 2024-05-09T03:02:29.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:02:29.571 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:29.639 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:29.640 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:29.640 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:29.640 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:29.671 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-09T03:02:29.674 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-09T03:02:29.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-05-09T03:02:29.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-09T03:02:29.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-09T03:02:29.893 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:29.893 INFO:tasks.ceph:role: osd.7 2024-05-09T03:02:29.893 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:02:29.893 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:02:29.957 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:29.958 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:29.990 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-09T03:02:29.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-09T03:02:29.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-05-09T03:02:30.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-09T03:02:30.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-05-09T03:02:30.195 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:30.195 INFO:tasks.ceph:role: osd.11 2024-05-09T03:02:30.195 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:02:30.195 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:02:30.268 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:30.268 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:30.268 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:30.269 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:30.300 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-09T03:02:30.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-09T03:02:30.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-05-09T03:02:30.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-05-09T03:02:30.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-05-09T03:02:30.507 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-05-09T03:02:30.507 INFO:tasks.ceph:role: osd.15 2024-05-09T03:02:30.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:02:30.508 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:02:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:02:30.573 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:02:30.573 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:02:30.604 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-09T03:02:30.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-09T03:02:30.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-05-09T03:02:30.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-05-09T03:02:30.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:02:30.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:30.842+0000 7f0858a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-09T03:02:33.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-09T03:02:33.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.805+0000 7f3480811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.805+0000 7f3480811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.805+0000 7f3480811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.805+0000 7f3480811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.805+0000 7f3480811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:02:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:33.806+0000 7f3480811640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-09T03:02:36.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-09T03:02:36.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:36.771 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-05-09T03:02:36.772 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-05-09T03:02:36.772 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-05-09T03:02:36.772 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-05-09T03:02:36.772 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-05-09T03:02:36.772 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:36.770+0000 7f94cc6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-05-09T03:02:39.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-05-09T03:02:39.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-05-09T03:02:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-09T03:02:39.733+0000 7f302820c640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-05-09T03:02:42.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-05-09T03:02:42.621 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-09T03:02:42.621 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:42.621 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-09T03:02:42.651 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:42.651 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-09T03:02:42.719 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:42.719 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-09T03:02:42.786 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:42.787 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-05-09T03:02:42.853 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:42.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-05-09T03:02:42.921 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:42.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-09T03:02:42.951 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:42.951 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-09T03:02:43.019 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:43.019 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-05-09T03:02:43.086 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:43.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-05-09T03:02:43.152 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:43.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-09T03:02:43.182 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:43.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-09T03:02:43.251 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:43.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-05-09T03:02:43.319 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:43.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-05-09T03:02:43.387 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-09T03:02:43.416 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-09T03:02:43.484 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-09T03:02:43.553 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-05-09T03:02:43.619 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-05-09T03:02:43.687 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:43.687 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-09T03:02:43.743 INFO:tasks.ceph:Adding keys to all mons... 2024-05-09T03:02:43.743 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T03:02:43.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T03:02:43.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T03:02:43.771 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.y] 2024-05-09T03:02:43.771 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCYPDxmN3WbGxAA6O+rdxKIx+shyw5zQsDmbQ== 2024-05-09T03:02:43.771 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCbPDxmWcOsNBAAYeMeyMYpkE3Qh/MJe730zQ== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.4] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCePDxmZ4R/MhAAcrDlOymQHcfdUCjnnts0cg== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.8] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQChPDxmx1KCMBAAQB9UrkMusmduwseUdwt1zw== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.12] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCkPDxmDRdmLhAAHRnWMYqxB5rHIvwoC38fxQ== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCqPDxmT3riLhAAFJOYcDMeTzECLFkM8qeH+A== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.5] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCtPDxmF9MaLhAAVIAG0C7xVH+gKnGPBu+esg== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.9] 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCwPDxmhQwmLBAAUQEQMfVzq2qiH7HwM4gNNA== 2024-05-09T03:02:43.772 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.13] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCzPDxma3NxKhAAxtim4tztRyJa4X3DWq/g8w== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQC5PDxmw/E2LRAAc5niAqRQBwIoqJcw/BVK2w== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.6] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQC8PDxmQvYcKxAADieSkdNQNTh+mxc6aiQ5AQ== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.10] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQC/PDxm3l8KKRAATHygwxbemrumBa8eTvQeWw== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.14] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDCPDxmshDyJhAAGvBVDmjtnGj5EfMTgM6zbA== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCYPDxmp37+HxAAZ/L9ubPI21lb/BTGA2ZLFw== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.3] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDGPDxmGVFJMhAAo5jWC5uHz+Dy6emhLCc6Mw== 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.7] 2024-05-09T03:02:43.773 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDJPDxmN1IXMBAAnUx0rnQ992VPrIhvli+oBw== 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.11] 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDMPDxmo8n5LRAAnxaTsd0aRQHqZXiHeeBykg== 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.15] 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDPPDxmefrIKxAAcQC+bdJ/KI7XRCOHBdChrQ== 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQCYPDxmjXy1IhAAjHt7kKI9qYU15Zo+0+s15g== 2024-05-09T03:02:43.774 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.y] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCYPDxmN3WbGxAA6O+rdxKIx+shyw5zQsDmbQ== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCbPDxmWcOsNBAAYeMeyMYpkE3Qh/MJe730zQ== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCePDxmZ4R/MhAAcrDlOymQHcfdUCjnnts0cg== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.8] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQChPDxmx1KCMBAAQB9UrkMusmduwseUdwt1zw== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.12] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCkPDxmDRdmLhAAHRnWMYqxB5rHIvwoC38fxQ== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCqPDxmT3riLhAAFJOYcDMeTzECLFkM8qeH+A== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCtPDxmF9MaLhAAVIAG0C7xVH+gKnGPBu+esg== 2024-05-09T03:02:43.775 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.9] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCwPDxmhQwmLBAAUQEQMfVzq2qiH7HwM4gNNA== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.13] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCzPDxma3NxKhAAxtim4tztRyJa4X3DWq/g8w== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC5PDxmw/E2LRAAc5niAqRQBwIoqJcw/BVK2w== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.6] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC8PDxmQvYcKxAADieSkdNQNTh+mxc6aiQ5AQ== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.10] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC/PDxm3l8KKRAATHygwxbemrumBa8eTvQeWw== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.14] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDCPDxmshDyJhAAGvBVDmjtnGj5EfMTgM6zbA== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCYPDxmp37+HxAAZ/L9ubPI21lb/BTGA2ZLFw== 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2024-05-09T03:02:43.776 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDGPDxmGVFJMhAAo5jWC5uHz+Dy6emhLCc6Mw== 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.7] 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDJPDxmN1IXMBAAnUx0rnQ992VPrIhvli+oBw== 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.11] 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDMPDxmo8n5LRAAnxaTsd0aRQHqZXiHeeBykg== 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.15] 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQDPPDxmefrIKxAAcQC+bdJ/KI7XRCOHBdChrQ== 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-09T03:02:43.777 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCYPDxmjXy1IhAAjHt7kKI9qYU15Zo+0+s15g== 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.y] 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCYPDxmN3WbGxAA6O+rdxKIx+shyw5zQsDmbQ== 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCbPDxmWcOsNBAAYeMeyMYpkE3Qh/MJe730zQ== 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.4] 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCePDxmZ4R/MhAAcrDlOymQHcfdUCjnnts0cg== 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.8] 2024-05-09T03:02:43.779 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQChPDxmx1KCMBAAQB9UrkMusmduwseUdwt1zw== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.12] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCkPDxmDRdmLhAAHRnWMYqxB5rHIvwoC38fxQ== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCqPDxmT3riLhAAFJOYcDMeTzECLFkM8qeH+A== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.5] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCtPDxmF9MaLhAAVIAG0C7xVH+gKnGPBu+esg== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.9] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCwPDxmhQwmLBAAUQEQMfVzq2qiH7HwM4gNNA== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.13] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCzPDxma3NxKhAAxtim4tztRyJa4X3DWq/g8w== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC5PDxmw/E2LRAAc5niAqRQBwIoqJcw/BVK2w== 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.6] 2024-05-09T03:02:43.780 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC8PDxmQvYcKxAADieSkdNQNTh+mxc6aiQ5AQ== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.10] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC/PDxm3l8KKRAATHygwxbemrumBa8eTvQeWw== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.14] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDCPDxmshDyJhAAGvBVDmjtnGj5EfMTgM6zbA== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCYPDxmp37+HxAAZ/L9ubPI21lb/BTGA2ZLFw== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDGPDxmGVFJMhAAo5jWC5uHz+Dy6emhLCc6Mw== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.7] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDJPDxmN1IXMBAAnUx0rnQ992VPrIhvli+oBw== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.11] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDMPDxmo8n5LRAAnxaTsd0aRQHqZXiHeeBykg== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.15] 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDPPDxmefrIKxAAcQC+bdJ/KI7XRCOHBdChrQ== 2024-05-09T03:02:43.781 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-05-09T03:02:43.782 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCYPDxmjXy1IhAAjHt7kKI9qYU15Zo+0+s15g== 2024-05-09T03:02:43.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:43.820 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:43.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:43.878 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.935 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.991 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.993 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:43.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.046 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.048 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.101 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.156 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.158 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.211 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.213 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.266 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.268 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.321 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.377 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.432 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.434 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.488 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.490 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.543 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:44.546 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:44.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:02:44.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.601 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.655 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.709 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.712 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.765 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:02:44.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:02:44.822 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:02:44.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:02:44.878 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-09T03:02:44.878 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-09T03:02:44.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:02:44.998 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-09T03:02:45.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-09T03:02:45.047 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:02:45.144 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-09T03:02:45.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-09T03:02:45.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:02:45.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-09T03:02:45.315 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:45.318 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:45.320 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:02:45.374 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-09T03:02:45.374 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T03:02:45.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T03:02:45.377 INFO:tasks.ceph.mon.a:Started 2024-05-09T03:02:45.377 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T03:02:45.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T03:02:45.379 INFO:tasks.ceph.mon.b:Started 2024-05-09T03:02:45.379 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T03:02:45.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T03:02:45.417 INFO:tasks.ceph.mon.c:Started 2024-05-09T03:02:45.417 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-09T03:02:45.417 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-09T03:02:45.417 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-09T03:02:45.419 INFO:tasks.ceph.mgr.y:Started 2024-05-09T03:02:45.419 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-09T03:02:45.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-09T03:02:45.422 INFO:tasks.ceph.mgr.x:Started 2024-05-09T03:02:45.422 INFO:tasks.ceph:Setting crush tunables to default 2024-05-09T03:02:45.423 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd crush tunables default 2024-05-09T03:02:55.792 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-05-09T03:02:55.805 INFO:tasks.ceph:check_enable_crimson: False 2024-05-09T03:02:55.805 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-09T03:02:55.805 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:55.805 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-09T03:02:55.829 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:55.829+0000 7f4fadc18040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T03:02:55.831 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:55.831 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-09T03:02:55.834 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:55.833+0000 7f4920e12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T03:02:55.896 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:55.896 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-05-09T03:02:55.961 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-05-09T03:02:55.961 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-05-09T03:02:55.992 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:55.991+0000 7f4920e12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:02:56.025 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:56.025 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-09T03:02:56.045 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:56.044+0000 7f4920e12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T03:02:56.054 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:56.055 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-09T03:02:56.058 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:56.058+0000 7f4fadc18040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:02:56.112 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:56.111+0000 7f4fadc18040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T03:02:56.127 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:56.128 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-05-09T03:02:56.196 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-09T03:02:56.196 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-05-09T03:02:56.263 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:56.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-09T03:02:56.293 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:56.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-09T03:02:56.298 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:56.297+0000 7f4920e12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T03:02:56.344 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:56.343+0000 7f4920e12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T03:02:56.364 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:56.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-05-09T03:02:56.372 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:56.371+0000 7f4fadc18040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T03:02:56.398 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-09T03:02:56.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-05-09T03:02:56.418 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:56.417+0000 7f4fadc18040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T03:02:56.469 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:56.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-09T03:02:56.499 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:56.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-09T03:02:56.573 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:56.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-05-09T03:02:56.639 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-09T03:02:56.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-05-09T03:02:56.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 8c1b94fb-7740-4feb-8b8f-fb5770e077d5 0 2024-05-09T03:02:56.930 INFO:teuthology.orchestra.run.smithi195.stdout:0 2024-05-09T03:02:56.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new e49dede2-db98-4c59-a34e-4d1b5f0e462f 1 2024-05-09T03:02:57.127 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-05-09T03:02:57.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new efedaad0-b417-4088-ae5d-01bfd0da36fd 2 2024-05-09T03:02:57.153 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.152+0000 7f4920e12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T03:02:57.200 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.199+0000 7f4fadc18040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T03:02:57.246 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.244+0000 7f4920e12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T03:02:57.290 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.288+0000 7f4920e12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T03:02:57.294 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.293+0000 7f4fadc18040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T03:02:57.326 INFO:teuthology.orchestra.run.smithi195.stdout:2 2024-05-09T03:02:57.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 5b55d622-7c34-4b8a-99a3-1aa3f20c542d 3 2024-05-09T03:02:57.338 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.337+0000 7f4fadc18040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T03:02:57.463 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.461+0000 7f4920e12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T03:02:57.503 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.502+0000 7f4fadc18040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T03:02:57.512 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.511+0000 7f4920e12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T03:02:57.549 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.548+0000 7f4fadc18040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T03:02:57.562 INFO:teuthology.orchestra.run.smithi195.stdout:3 2024-05-09T03:02:57.565 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.564+0000 7f4920e12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T03:02:57.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new c287eca5-ba95-4790-851b-b2a9a428f2e0 4 2024-05-09T03:02:57.602 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.601+0000 7f4fadc18040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T03:02:57.683 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:57.682+0000 7f4920e12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:02:57.717 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:57.716+0000 7f4fadc18040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:02:57.750 INFO:teuthology.orchestra.run.smithi195.stdout:4 2024-05-09T03:02:57.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc 5 2024-05-09T03:02:57.939 INFO:teuthology.orchestra.run.smithi195.stdout:5 2024-05-09T03:02:57.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 2c63dad7-284d-4ca4-80a9-fa6bdd53e708 6 2024-05-09T03:02:58.076 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.075+0000 7f4920e12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T03:02:58.099 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.098+0000 7f4fadc18040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T03:02:58.129 INFO:teuthology.orchestra.run.smithi195.stdout:6 2024-05-09T03:02:58.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new accddb1d-46ac-4197-a964-4375fc9b64dc 7 2024-05-09T03:02:58.219 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.218+0000 7f4920e12040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T03:02:58.237 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.236+0000 7f4fadc18040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T03:02:58.320 INFO:teuthology.orchestra.run.smithi195.stdout:7 2024-05-09T03:02:58.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 23691442-90a0-49ad-a2eb-431283762f93 8 2024-05-09T03:02:58.489 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.488+0000 7f4920e12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T03:02:58.498 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.497+0000 7f4fadc18040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T03:02:58.511 INFO:teuthology.orchestra.run.smithi195.stdout:8 2024-05-09T03:02:58.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2 9 2024-05-09T03:02:58.544 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.543+0000 7f4920e12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T03:02:58.551 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.550+0000 7f4fadc18040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T03:02:58.664 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.663+0000 7f4920e12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T03:02:58.669 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.668+0000 7f4fadc18040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T03:02:58.700 INFO:teuthology.orchestra.run.smithi195.stdout:9 2024-05-09T03:02:58.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new cb824145-f6c1-4840-8e96-7e7a93eb7632 10 2024-05-09T03:02:58.929 INFO:teuthology.orchestra.run.smithi195.stdout:10 2024-05-09T03:02:58.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new d99a7a94-04b1-459d-a876-83922eee12e1 11 2024-05-09T03:02:58.957 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:58.956+0000 7f4fadc18040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T03:02:58.973 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:58.972+0000 7f4920e12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T03:02:59.004 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.003+0000 7f4fadc18040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T03:02:59.022 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.021+0000 7f4920e12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T03:02:59.120 INFO:teuthology.orchestra.run.smithi195.stdout:11 2024-05-09T03:02:59.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new c49d43a8-5c37-498a-9f19-0606d7e1c650 12 2024-05-09T03:02:59.158 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.157+0000 7f4fadc18040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T03:02:59.184 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.183+0000 7f4920e12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T03:02:59.293 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.292+0000 7f4fadc18040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T03:02:59.325 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.323+0000 7f4920e12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T03:02:59.341 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.341+0000 7f4fadc18040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T03:02:59.349 INFO:teuthology.orchestra.run.smithi195.stdout:12 2024-05-09T03:02:59.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 92ba1257-5f50-4ab2-a50b-d271cbcb0d3c 13 2024-05-09T03:02:59.376 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.375+0000 7f4920e12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T03:02:59.411 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.410+0000 7f4fadc18040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T03:02:59.449 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.448+0000 7f4920e12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T03:02:59.458 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.454+0000 7f4fadc18040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T03:02:59.497 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.495+0000 7f4920e12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T03:02:59.554 INFO:teuthology.orchestra.run.smithi195.stdout:13 2024-05-09T03:02:59.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new b69b003c-cea6-42bc-9744-77b883680460 14 2024-05-09T03:02:59.744 INFO:teuthology.orchestra.run.smithi195.stdout:14 2024-05-09T03:02:59.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new dc295c88-8112-46dd-a3b2-b9fb217220bb 15 2024-05-09T03:02:59.930 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:02:59.929+0000 7f4fadc18040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T03:02:59.936 INFO:teuthology.orchestra.run.smithi195.stdout:15 2024-05-09T03:02:59.947 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-09T03:02:59.948 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-09T03:02:59.950 INFO:tasks.ceph.osd.0:Started 2024-05-09T03:02:59.950 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-09T03:02:59.950 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-09T03:02:59.952 INFO:tasks.ceph.osd.4:Started 2024-05-09T03:02:59.952 INFO:tasks.ceph.osd.8:Restarting daemon 2024-05-09T03:02:59.953 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-05-09T03:02:59.955 INFO:tasks.ceph.osd.8:Started 2024-05-09T03:02:59.955 INFO:tasks.ceph.osd.12:Restarting daemon 2024-05-09T03:02:59.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-05-09T03:02:59.957 INFO:tasks.ceph.osd.12:Started 2024-05-09T03:02:59.957 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-09T03:02:59.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-09T03:02:59.960 INFO:tasks.ceph.osd.1:Started 2024-05-09T03:02:59.960 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-09T03:02:59.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-09T03:02:59.963 INFO:tasks.ceph.osd.5:Started 2024-05-09T03:02:59.963 INFO:tasks.ceph.osd.9:Restarting daemon 2024-05-09T03:02:59.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-05-09T03:02:59.965 INFO:tasks.ceph.osd.9:Started 2024-05-09T03:02:59.965 INFO:tasks.ceph.osd.13:Restarting daemon 2024-05-09T03:02:59.965 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-05-09T03:02:59.968 INFO:tasks.ceph.osd.13:Started 2024-05-09T03:02:59.968 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-09T03:02:59.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-09T03:02:59.971 INFO:tasks.ceph.osd.2:Started 2024-05-09T03:02:59.971 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-09T03:02:59.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-09T03:02:59.973 INFO:tasks.ceph.osd.6:Started 2024-05-09T03:02:59.973 INFO:tasks.ceph.osd.10:Restarting daemon 2024-05-09T03:02:59.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-05-09T03:02:59.975 INFO:tasks.ceph.osd.10:Started 2024-05-09T03:02:59.976 INFO:tasks.ceph.osd.14:Restarting daemon 2024-05-09T03:02:59.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-05-09T03:02:59.978 INFO:tasks.ceph.osd.14:Started 2024-05-09T03:02:59.978 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-09T03:02:59.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-09T03:02:59.980 INFO:tasks.ceph.osd.3:Started 2024-05-09T03:02:59.980 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-09T03:02:59.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-09T03:02:59.982 INFO:tasks.ceph.osd.7:Started 2024-05-09T03:02:59.983 INFO:tasks.ceph.osd.11:Restarting daemon 2024-05-09T03:02:59.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-05-09T03:02:59.985 INFO:tasks.ceph.osd.11:Started 2024-05-09T03:02:59.985 INFO:tasks.ceph.osd.15:Restarting daemon 2024-05-09T03:02:59.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-05-09T03:02:59.987 INFO:tasks.ceph.osd.15:Started 2024-05-09T03:02:59.987 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:02:59.989 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:02:59.988+0000 7f4920e12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T03:03:00.080 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:03:00.079+0000 7f4fadc18040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T03:03:00.130 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-05-09T03:03:00.129+0000 7f4920e12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T03:03:00.183 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-05-09T03:03:00.183 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:00.194 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-09T03:03:00.194 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-09T03:03:00.550 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-05-09T03:03:00.549+0000 7fc6a761f640 -1 Falling back to public interface 2024-05-09T03:03:00.568 INFO:tasks.ceph.osd.8.smithi072.stderr:2024-05-09T03:03:00.567+0000 7ffaed89c640 -1 Falling back to public interface 2024-05-09T03:03:00.573 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-05-09T03:03:00.572+0000 7f0924c0e640 -1 Falling back to public interface 2024-05-09T03:03:00.578 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-05-09T03:03:00.577+0000 7f9cad799640 -1 Falling back to public interface 2024-05-09T03:03:00.581 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-09T03:03:00.580+0000 7f1a00a11640 -1 Falling back to public interface 2024-05-09T03:03:00.582 INFO:tasks.ceph.osd.9.smithi096.stderr:2024-05-09T03:03:00.580+0000 7f0036142640 -1 Falling back to public interface 2024-05-09T03:03:00.582 INFO:tasks.ceph.osd.9.smithi096.stderr:2024-05-09T03:03:00.580+0000 7f00350f1640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2024-05-09T03:03:00.582 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-05-09T03:03:00.581+0000 7ff041b8e640 -1 Falling back to public interface 2024-05-09T03:03:00.583 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-05-09T03:03:00.581+0000 7f6b39021640 -1 Falling back to public interface 2024-05-09T03:03:00.583 INFO:tasks.ceph.osd.12.smithi072.stderr:2024-05-09T03:03:00.581+0000 7f98bdf6e640 -1 Falling back to public interface 2024-05-09T03:03:00.583 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-05-09T03:03:00.581+0000 7ff040b3d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-05-09T03:03:00.584 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-05-09T03:03:00.582+0000 7fad95614640 -1 Falling back to public interface 2024-05-09T03:03:00.584 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-05-09T03:03:00.583+0000 7f05ce211640 -1 Falling back to public interface 2024-05-09T03:03:00.585 INFO:tasks.ceph.osd.13.smithi096.stderr:2024-05-09T03:03:00.584+0000 7fa74c418640 -1 Falling back to public interface 2024-05-09T03:03:00.587 INFO:tasks.ceph.osd.10.smithi181.stderr:2024-05-09T03:03:00.586+0000 7f66f1cae640 -1 Falling back to public interface 2024-05-09T03:03:00.596 INFO:tasks.ceph.osd.14.smithi181.stderr:2024-05-09T03:03:00.595+0000 7fe86ae2e640 -1 Falling back to public interface 2024-05-09T03:03:00.598 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-05-09T03:03:00.596+0000 7fdea241a640 -1 Falling back to public interface 2024-05-09T03:03:00.604 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-05-09T03:03:00.603+0000 7fc3ca415640 -1 Falling back to public interface 2024-05-09T03:03:01.095 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:03:01.093+0000 7f4f4d42c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-09T03:03:01.095 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-05-09T03:03:01.094+0000 7f4f4d42c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-09T03:03:01.806 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:02.104 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:02.105 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:02.116 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:03.986 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-05-09T03:03:03.986+0000 7fc6a761f640 -1 osd.4 0 log_to_monitors true 2024-05-09T03:03:04.028 INFO:tasks.ceph.osd.8.smithi072.stderr:2024-05-09T03:03:04.027+0000 7ffaed89c640 -1 osd.8 0 log_to_monitors true 2024-05-09T03:03:04.044 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-05-09T03:03:04.043+0000 7f05ce211640 -1 osd.6 0 log_to_monitors true 2024-05-09T03:03:04.055 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-05-09T03:03:04.054+0000 7f9cad799640 -1 osd.7 0 log_to_monitors true 2024-05-09T03:03:04.058 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-05-09T03:03:04.057+0000 7f0924c0e640 -1 osd.5 0 log_to_monitors true 2024-05-09T03:03:04.060 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-05-09T03:03:04.059+0000 7fad95614640 -1 osd.11 0 log_to_monitors true 2024-05-09T03:03:04.062 INFO:tasks.ceph.osd.9.smithi096.stderr:2024-05-09T03:03:04.061+0000 7f0036142640 -1 osd.9 0 log_to_monitors true 2024-05-09T03:03:04.062 INFO:tasks.ceph.osd.12.smithi072.stderr:2024-05-09T03:03:04.061+0000 7f98bdf6e640 -1 osd.12 0 log_to_monitors true 2024-05-09T03:03:04.074 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-05-09T03:03:04.073+0000 7ff041b8e640 -1 osd.0 0 log_to_monitors true 2024-05-09T03:03:04.090 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-05-09T03:03:04.090+0000 7f6b39021640 -1 osd.2 0 log_to_monitors true 2024-05-09T03:03:04.092 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-09T03:03:04.091+0000 7f1a00a11640 -1 osd.1 0 log_to_monitors true 2024-05-09T03:03:04.098 INFO:tasks.ceph.osd.10.smithi181.stderr:2024-05-09T03:03:04.098+0000 7f66f1cae640 -1 osd.10 0 log_to_monitors true 2024-05-09T03:03:04.101 INFO:tasks.ceph.osd.13.smithi096.stderr:2024-05-09T03:03:04.100+0000 7fa74c418640 -1 osd.13 0 log_to_monitors true 2024-05-09T03:03:04.102 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-05-09T03:03:04.101+0000 7fc3ca415640 -1 osd.3 0 log_to_monitors true 2024-05-09T03:03:04.103 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-05-09T03:03:04.101+0000 7fdea241a640 -1 osd.15 0 log_to_monitors true 2024-05-09T03:03:04.114 INFO:tasks.ceph.osd.14.smithi181.stderr:2024-05-09T03:03:04.114+0000 7fe86ae2e640 -1 osd.14 0 log_to_monitors true 2024-05-09T03:03:09.728 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:10.023 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:10.023 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:10.035 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:17.651 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:17.948 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:17.949 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:17.960 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:25.574 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:25.869 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:25.869 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:25.881 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:33.496 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:33.792 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:33.792 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:33.804 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:41.419 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:41.715 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:41.715 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:41.727 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:49.341 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:49.637 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:49.638 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:49.649 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:03:57.263 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:03:57.559 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:03:57.559 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:03:57.571 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:05.186 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:05.484 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:05.485 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:05.496 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:13.113 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:13.411 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:13.412 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:13.423 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:21.036 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:21.332 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:21.332 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:21.344 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:28.958 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:29.253 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:29.254 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:29.265 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:36.880 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:37.175 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:37.176 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:37.187 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:44.801 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:45.097 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:45.098 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:45.109 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:04:52.724 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:04:53.020 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:04:53.020 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:04:53.032 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:00.647 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:00.942 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:00.943 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:00.954 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:08.569 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:08.864 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:08.864 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:08.876 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:16.490 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:16.787 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:16.788 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:16.799 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:24.413 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:24.710 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:24.710 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:24.722 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:32.335 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:32.628 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:32.628 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:32.640 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:40.253 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:40.548 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:40.549 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:40.560 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:48.175 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:48.470 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:48.470 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:48.482 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:05:56.095 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:05:56.397 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:05:56.397 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:05:56.409 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:04.023 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:04.323 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:04.323 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:04.335 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:11.947 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:12.246 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:12.246 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:12.257 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:19.872 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:20.169 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:20.169 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:20.181 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:27.795 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:28.092 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:28.092 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:28.103 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:35.717 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:36.012 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:36.012 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:36.024 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:43.636 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:43.935 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:43.936 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:43.947 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:51.560 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:51.855 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:51.855 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:51.867 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:06:59.480 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:06:59.777 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:06:59.777 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:06:59.789 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:07.403 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:07.702 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:07.702 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:07.714 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:15.328 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:15.628 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:15.628 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:15.639 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:23.253 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:23.549 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:23.549 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:23.560 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:31.175 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:31.471 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:31.472 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:31.483 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:39.098 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:39.392 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:39.392 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:39.404 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:47.019 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:47.315 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:47.315 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:47.327 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:07:54.941 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:07:55.238 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:07:55.239 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:07:55.251 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:02.867 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:03.165 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:03.165 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:03.176 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:10.792 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:11.085 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:11.086 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:11.097 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:18.711 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:19.007 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:19.007 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:19.019 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:26.633 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:26.929 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:26.929 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:26.941 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:34.555 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:34.851 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:34.851 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:34.862 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:42.475 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:42.773 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:42.773 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:42.784 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:50.398 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:50.698 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:50.698 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:50.710 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:08:58.324 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:08:58.621 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:08:58.621 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:08:58.634 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:06.249 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:06.542 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:06.542 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:06.554 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:14.169 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:14.466 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:14.466 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:14.478 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:22.091 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:22.388 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:22.388 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:22.400 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:30.015 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:30.316 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:30.316 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:30.327 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:37.942 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:38.241 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:38.241 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:38.253 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:45.868 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:46.165 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:46.165 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:46.177 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:09:53.792 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:54.087 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:09:54.087 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:54.099 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:01.711 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:02.008 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:02.009 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:02.020 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:09.635 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:09.929 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:09.929 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:09.941 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:17.556 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:17.852 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:17.852 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:17.863 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:25.478 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:25.778 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:25.778 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:25.790 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:33.404 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:33.698 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:33.699 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:33.710 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:41.326 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:41.621 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:41.621 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:41.633 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:49.248 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:49.548 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:49.548 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:49.557 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:10:57.171 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:57.467 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:10:57.467 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:57.480 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:05.095 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:05.392 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:05.392 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:05.404 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:13.019 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:13.315 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:13.315 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:13.327 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:20.941 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:21.238 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:21.238 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:21.250 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:28.863 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:29.160 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:29.160 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:29.172 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:36.785 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:37.084 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:37.084 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:37.096 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:44.709 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:45.007 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:45.007 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:45.019 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:11:52.633 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:52.934 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:11:52.934 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:52.946 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:00.560 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:00.854 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:00.854 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:00.866 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:08.482 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:08.778 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:08.778 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:08.789 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:16.403 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:16.701 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:16.702 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:16.713 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:24.323 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:24.619 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:24.619 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:24.631 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:32.244 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:32.541 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:32.541 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:32.553 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:40.168 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:40.464 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:40.464 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:40.475 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:48.090 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:48.388 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:48.388 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:48.399 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:12:56.015 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:56.312 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:12:56.312 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":18,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:02:59.933211+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:02:59.933211+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:56.324 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:03.938 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:04.236 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:04.236 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:04.247 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:11.862 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:12.158 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:12.158 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:12.169 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:19.783 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:20.078 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:20.078 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:20.089 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:27.703 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:28.000 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:28.001 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:28.012 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:35.626 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:35.927 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:35.927 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:35.936 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:43.550 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:43.846 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:43.846 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:43.858 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:51.473 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:51.769 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:51.769 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:51.780 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:13:59.393 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:13:59.691 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:13:59.691 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:13:59.702 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:07.316 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:07.611 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:07.612 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:07.623 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:15.237 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:15.535 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:15.535 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:15.547 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:23.161 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:23.457 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:23.457 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:23.468 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:31.082 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:31.383 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:31.383 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:31.394 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:39.009 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:39.306 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:39.306 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:39.318 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:46.933 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:47.228 INFO:teuthology.misc.health.smithi072.stdout: 2024-05-09T03:14:47.228 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":19,"fsid":"f061f132-a796-4bcc-9d93-8f60c6bd3094","created":"2024-05-09T03:02:55.682202+0000","modified":"2024-05-09T03:13:00.550678+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:13:00.550678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8c1b94fb-7740-4feb-8b8f-fb5770e077d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e49dede2-db98-4c59-a34e-4d1b5f0e462f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"efedaad0-b417-4088-ae5d-01bfd0da36fd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5b55d622-7c34-4b8a-99a3-1aa3f20c542d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":4,"uuid":"c287eca5-ba95-4790-851b-b2a9a428f2e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":5,"uuid":"1e15ceb0-9ca8-4d16-88b5-26bb98f9c0dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c63dad7-284d-4ca4-80a9-fa6bdd53e708","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"accddb1d-46ac-4197-a964-4375fc9b64dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"23691442-90a0-49ad-a2eb-431283762f93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"bb63cd18-272e-4f98-9bc1-b2cf1d1e1fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb824145-f6c1-4840-8e96-7e7a93eb7632","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"d99a7a94-04b1-459d-a876-83922eee12e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"c49d43a8-5c37-498a-9f19-0606d7e1c650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"92ba1257-5f50-4ab2-a50b-d271cbcb0d3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"b69b003c-cea6-42bc-9744-77b883680460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"dc295c88-8112-46dd-a3b2-b9fb217220bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:47.240 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-05-09T03:14:47.240 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-05-09T03:14:47.241 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-09T03:14:47.242 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-09T03:14:47.242 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.275 INFO:tasks.ceph.osd.0:Stopped 2024-05-09T03:14:47.275 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-09T03:14:47.275 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.342 INFO:tasks.ceph.osd.4:Stopped 2024-05-09T03:14:47.343 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-05-09T03:14:47.343 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.407 INFO:tasks.ceph.osd.8:Stopped 2024-05-09T03:14:47.407 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-05-09T03:14:47.407 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.473 INFO:tasks.ceph.osd.12:Stopped 2024-05-09T03:14:47.473 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-09T03:14:47.473 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.508 INFO:tasks.ceph.osd.1:Stopped 2024-05-09T03:14:47.508 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-09T03:14:47.508 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.575 INFO:tasks.ceph.osd.5:Stopped 2024-05-09T03:14:47.575 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-05-09T03:14:47.575 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.642 INFO:tasks.ceph.osd.9:Stopped 2024-05-09T03:14:47.642 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-05-09T03:14:47.642 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.710 INFO:tasks.ceph.osd.13:Stopped 2024-05-09T03:14:47.711 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-09T03:14:47.711 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.746 INFO:tasks.ceph.osd.2:Stopped 2024-05-09T03:14:47.746 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-09T03:14:47.746 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.821 INFO:tasks.ceph.osd.6:Stopped 2024-05-09T03:14:47.821 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-05-09T03:14:47.822 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.894 INFO:tasks.ceph.osd.10:Stopped 2024-05-09T03:14:47.895 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-05-09T03:14:47.895 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:47.965 INFO:tasks.ceph.osd.14:Stopped 2024-05-09T03:14:47.965 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-09T03:14:47.965 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.000 INFO:tasks.ceph.osd.3:Stopped 2024-05-09T03:14:48.000 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-09T03:14:48.000 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.067 INFO:tasks.ceph.osd.7:Stopped 2024-05-09T03:14:48.067 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-05-09T03:14:48.067 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.140 INFO:tasks.ceph.osd.11:Stopped 2024-05-09T03:14:48.140 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-05-09T03:14:48.140 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.212 INFO:tasks.ceph.osd.15:Stopped 2024-05-09T03:14:48.213 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-09T03:14:48.213 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-09T03:14:48.213 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.243 INFO:tasks.ceph.mgr.y:Stopped 2024-05-09T03:14:48.243 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-09T03:14:48.243 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.283 INFO:tasks.ceph.mgr.x:Stopped 2024-05-09T03:14:48.283 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-09T03:14:48.283 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T03:14:48.283 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.297 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T03:14:48.297 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T03:14:48.297 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.310 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T03:14:48.311 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T03:14:48.311 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.324 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T03:14:48.324 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-09T03:14:48.324 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'must scrub before tier agent can activate' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-05-09T03:14:48.375 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:14:48.375 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-09T03:14:48.564 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:14:48.565 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-09T03:14:48.648 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:14:48.648 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-05-09T03:14:48.713 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:14:48.713 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-05-09T03:14:48.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:14:48.771 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-09T03:14:48.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:14:48.888 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-09T03:14:48.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:14:48.947 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-05-09T03:14:49.006 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:14:49.006 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-05-09T03:14:49.056 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:14:49.056 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-09T03:14:49.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:14:49.173 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-09T03:14:49.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:14:49.251 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-05-09T03:14:49.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:14:49.301 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-05-09T03:14:49.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:14:49.349 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-09T03:14:49.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:14:49.453 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-09T03:14:49.532 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:14:49.533 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-05-09T03:14:49.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:14:49.591 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-05-09T03:14:49.647 INFO:tasks.ceph:Archiving mon data... 2024-05-09T03:14:49.649 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/data/mon.a.tgz 2024-05-09T03:14:49.649 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-05-09T03:14:49.668 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.qFVtMpEzKw 2024-05-09T03:14:49.669 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qFVtMpEzKw 2024-05-09T03:14:49.778 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.qFVtMpEzKw 2024-05-09T03:14:49.864 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.qFVtMpEzKw is 138KB 2024-05-09T03:14:49.889 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.qFVtMpEzKw 2024-05-09T03:14:49.904 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/data/mon.b.tgz 2024-05-09T03:14:49.904 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-05-09T03:14:49.923 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.IcQHzDoecG 2024-05-09T03:14:49.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.IcQHzDoecG 2024-05-09T03:14:50.024 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.IcQHzDoecG 2024-05-09T03:14:50.111 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.IcQHzDoecG is 139KB 2024-05-09T03:14:50.166 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.IcQHzDoecG 2024-05-09T03:14:50.181 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/data/mon.c.tgz 2024-05-09T03:14:50.181 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-05-09T03:14:50.200 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.WlL4syPQLC 2024-05-09T03:14:50.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.WlL4syPQLC 2024-05-09T03:14:50.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.WlL4syPQLC 2024-05-09T03:14:50.397 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.WlL4syPQLC is 139KB 2024-05-09T03:14:50.422 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.WlL4syPQLC 2024-05-09T03:14:50.438 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-09T03:14:50.438 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:14:50.441 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:14:50.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:14:50.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:14:50.530 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:14:50.546 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:14:50.562 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:14:50.580 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:14:50.580 INFO:tasks.ceph:Archiving crash dumps... 2024-05-09T03:14:50.582 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi072/crash 2024-05-09T03:14:50.582 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:14:50.611 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi096/crash 2024-05-09T03:14:50.611 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:14:50.637 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi181/crash 2024-05-09T03:14:50.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:14:50.664 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi195/crash 2024-05-09T03:14:50.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:14:50.702 INFO:tasks.ceph:Compressing logs... 2024-05-09T03:14:50.702 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:14:50.705 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:14:50.708 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:14:50.710 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-09T03:14:50.731 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-09T03:14:50.732 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.5.log: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T03:14:50.732 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.1.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T03:14:50.732 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-05-09T03:14:50.736 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T03:14:50.736 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-09T03:14:50.736 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T03:14:50.736 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-09T03:14:50.737 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-09T03:14:50.737 INFO:teuthology.orchestra.run.smithi181.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T03:14:50.737 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-05-09T03:14:50.737 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.c.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T03:14:50.737 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36645.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37859.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38576.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36645.log.gz 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38811.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37859.log.gz 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38576.log.gz 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38811.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38811.log.gz 2024-05-09T03:14:50.781 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26445.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38850.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.tmp-client.admin.26445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26445.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38850.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38655.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-05-09T03:14:50.782 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36489.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36606.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38419.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36489.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37155.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36606.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38419.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36177.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37155.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36411.log 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-05-09T03:14:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37938.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36177.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37389.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36411.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38537.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37938.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37389.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38537.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37116.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37781.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-05-09T03:14:50.784 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37781.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36372.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36372.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38068.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36255.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36255.log.gz 2024-05-09T03:14:50.785 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36684.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38029.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38107.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36684.log.gz 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36879.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38029.log.gz 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36294.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38107.log.gz 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36879.log.gz 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-05-09T03:14:50.786 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36294.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36060.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38459.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36060.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36060.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-mon.a.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38459.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37898.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.35982.log 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-05-09T03:14:50.787 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36567.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.35982.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35982.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36998.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36567.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36450.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36998.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36998.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38772.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36450.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37625.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38772.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38772.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37547.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37625.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37547.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37547.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37990.log 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-05-09T03:14:50.788 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37990.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37990.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36762.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37703.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36762.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35865.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37703.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36528.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.35865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35865.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37467.log 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36528.log.gz 2024-05-09T03:14:50.789 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37467.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.35943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39045.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39045.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36801.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36801.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36099.log 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-05-09T03:14:50.790 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36099.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37233.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37311.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36333.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37233.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32542.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37311.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36333.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38733.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.32542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32542.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36959.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38146.log 2024-05-09T03:14:50.791 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36959.log: /var/log/ceph/ceph-client.admin.38733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38733.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36959.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38185.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38146.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37037.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38185.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37037.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37350.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37037.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36723.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36920.log: /var/log/ceph/ceph-client.admin.37350.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37350.log.gz 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32744.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36021.log 2024-05-09T03:14:50.792 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36723.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.32744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32744.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.32275.log: /var/log/ceph/ceph-client.admin.36021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36021.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35904.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.35904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35904.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37586.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37272.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37586.log: /var/log/ceph/ceph-client.admin.37742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37586.log.gz 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38616.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36840.log 2024-05-09T03:14:50.793 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37272.log.gz 2024-05-09T03:14:50.794 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38616.log.gz 2024-05-09T03:14:50.794 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.36840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36840.log.gz 2024-05-09T03:14:50.795 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph-osd.14.log: 96.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-09T03:14:50.795 INFO:teuthology.orchestra.run.smithi181.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-05-09T03:14:50.795 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.9.log: 96.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-09T03:14:50.795 INFO:teuthology.orchestra.run.smithi096.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-09T03:14:50.795 INFO:teuthology.orchestra.run.smithi096.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32395.log 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31955.log 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32395.log.gz 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32355.log 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.31955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32475.log 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31955.log.gz 2024-05-09T03:14:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32355.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32355.log.gz -5 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32235.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32435.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32475.log.gz 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32235.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32235.log.gz 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32435.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32435.log.gz 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32035.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32035.log.gz 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32115.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31915.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.32115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32115.log.gz 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32155.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32195.log 2024-05-09T03:14:50.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.31915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31915.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32155.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32315.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32195.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32075.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32315.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32315.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr: -- /var/log/ceph/ceph-client.admin.31995.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32075.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.31995.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31995.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31866.log 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.32275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.31866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31866.log.gz 2024-05-09T03:14:50.804 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-09T03:14:50.805 INFO:teuthology.orchestra.run.smithi096.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-05-09T03:14:50.805 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-09T03:14:50.805 INFO:teuthology.orchestra.run.smithi072.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-09T03:14:50.806 INFO:teuthology.orchestra.run.smithi181.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-09T03:14:50.807 INFO:teuthology.orchestra.run.smithi181.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-05-09T03:14:50.810 INFO:teuthology.orchestra.run.smithi072.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-05-09T03:14:50.812 INFO:teuthology.orchestra.run.smithi072.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-09T03:14:50.826 INFO:teuthology.orchestra.run.smithi195.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-05-09T03:14:50.826 INFO:teuthology.orchestra.run.smithi072.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-05-09T03:14:50.828 INFO:teuthology.orchestra.run.smithi195.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-09T03:14:50.830 INFO:teuthology.orchestra.run.smithi195.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-05-09T03:14:50.830 INFO:teuthology.orchestra.run.smithi195.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-09T03:14:50.833 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-09T03:14:50.833 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.067s 2024-05-09T03:14:50.833 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m0.214s 2024-05-09T03:14:50.833 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.034s 2024-05-09T03:14:50.854 INFO:teuthology.orchestra.run.smithi181.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-09T03:14:50.856 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-05-09T03:14:50.856 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m0.134s 2024-05-09T03:14:50.856 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m0.356s 2024-05-09T03:14:50.856 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.032s 2024-05-09T03:14:50.858 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-09T03:14:50.861 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-09T03:14:50.861 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m0.140s 2024-05-09T03:14:50.861 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m0.342s 2024-05-09T03:14:50.861 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.026s 2024-05-09T03:14:51.016 INFO:teuthology.orchestra.run.smithi072.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-09T03:14:51.017 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-05-09T03:14:51.018 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m0.298s 2024-05-09T03:14:51.018 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m0.613s 2024-05-09T03:14:51.018 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m0.090s 2024-05-09T03:14:51.018 INFO:tasks.ceph:Archiving logs... 2024-05-09T03:14:51.018 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi072/log 2024-05-09T03:14:51.019 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:14:51.235 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi096/log 2024-05-09T03:14:51.235 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:14:51.301 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi181/log 2024-05-09T03:14:51.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:14:51.373 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi195/log 2024-05-09T03:14:51.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:14:51.439 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-05-09T03:14:51.642 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=325d9143fe8947ab8a2571e7cd325337 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c152e9f40d11b517c5526b6e8144d52000d7e690/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-05-09T03:14:51.645 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-09T03:14:51.654 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T03:14:51.673 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T03:14:51.674 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:14:51.688 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:14:51.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:14:51.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:14:51.807 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T03:14:51.815 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T03:14:51.842 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:14:51.845 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:14:51.847 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:14:51.850 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:14:51.857 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:14:51.861 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 +2778us[+2793us] +/- 43ms 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 -2217us[-2217us] +/- 78ms 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -2609us[-2609us] +/- 79ms 2024-05-09T03:14:51.862 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:14:51.863 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:14:51.866 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:14:51.866 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-05-09T03:14:51.866 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 +2846us[+2853us] +/- 43ms 2024-05-09T03:14:51.866 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 -2128us[-2128us] +/- 78ms 2024-05-09T03:14:51.867 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -2525us[-2525us] +/- 79ms 2024-05-09T03:14:51.867 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:14:51.867 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:14:51.868 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:14:51.869 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-05-09T03:14:51.869 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +2764us[+2781us] +/- 43ms 2024-05-09T03:14:51.869 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -2225us[-2225us] +/- 78ms 2024-05-09T03:14:51.869 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -2621us[-2621us] +/- 79ms 2024-05-09T03:14:51.869 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:14:51.872 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:14:51.872 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-05-09T03:14:51.872 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +2761us[+2778us] +/- 43ms 2024-05-09T03:14:51.872 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -2224us[-2208us] +/- 78ms 2024-05-09T03:14:51.873 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -2603us[-2587us] +/- 79ms 2024-05-09T03:14:51.873 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:14:51.874 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T03:14:51.882 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T03:14:51.882 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T03:14:51.890 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:14:51.964 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:14:52.030 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:14:52.090 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:14:52.155 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:14:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:52.183 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:14:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:52.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:14:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:52.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:14:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:52.269 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T03:14:52.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T03:14:52.292 INFO:teuthology.task.internal:Duration was 1197.697412 seconds 2024-05-09T03:14:52.292 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T03:14:52.301 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T03:14:52.301 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:14:52.304 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:14:52.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:14:52.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:14:52.341 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:14:52.346 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:14:52.349 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:14:52.352 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:14:52.832 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T03:14:52.833 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-05-09T03:14:52.833 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:14:52.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-05-09T03:14:52.854 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:14:52.883 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-05-09T03:14:52.884 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:14:52.915 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-05-09T03:14:52.915 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:14:52.950 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T03:14:52.950 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:14:52.953 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:14:52.955 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:14:52.958 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:14:53.020 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T03:14:53.021 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:14:53.024 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:14:53.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:14:53.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:14:53.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T03:14:53.166 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T03:14:53.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:14:53.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:14:53.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:14:53.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:14:53.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T03:14:53.276 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.279 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.299 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-05-09T03:14:53.302 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-05-09T03:14:53.305 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-05-09T03:14:53.330 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-05-09T03:14:53.351 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:53.375 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:53.391 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:53.405 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:14:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:14:53.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T03:14:53.431 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T03:14:53.431 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi072 2024-05-09T03:14:53.432 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:14:53.465 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi096 2024-05-09T03:14:53.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:14:53.499 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi181 2024-05-09T03:14:53.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:14:53.533 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454/remote/smithi195 2024-05-09T03:14:53.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:14:53.565 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T03:14:53.565 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:14:53.568 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:14:53.570 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:14:53.573 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:14:53.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T03:14:53.627 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T03:14:53.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T03:14:53.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T03:14:53.639 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:14:53.641 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:14:53.643 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:14:53.646 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:14:53.654 INFO:teuthology.orchestra.run.smithi072.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:14 /home/ubuntu/cephtest 2024-05-09T03:14:53.656 INFO:teuthology.orchestra.run.smithi096.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:14 /home/ubuntu/cephtest 2024-05-09T03:14:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:14 /home/ubuntu/cephtest 2024-05-09T03:14:53.670 INFO:teuthology.orchestra.run.smithi195.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:14 /home/ubuntu/cephtest 2024-05-09T03:14:53.672 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T03:14:53.686 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T03:14:53.714 INFO:teuthology.nuke:Checking targets against current locks 2024-05-09T03:14:53.741 DEBUG:teuthology.nuke:shortname: smithi072 2024-05-09T03:14:53.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T03:14:53.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.130185', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAz2REg4lMucipWe/Qym4OExMaGQThHxYzzcoO5SnYyhhtPFTUBV1VOFUK4Im5i5selAYjBOh5T04h7agwIORdE='} 2024-05-09T03:14:53.772 DEBUG:teuthology.nuke:shortname: smithi096 2024-05-09T03:14:53.772 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T03:14:53.797 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.128609', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEEG9lnjv2iuYW/yBzKLhScFVmU519RnxmwnqXsoMDNaGITzCPfZU3OoyDf6nQ2Q4iPsTD2D9CbR6lllHxsPs0='} 2024-05-09T03:14:53.803 DEBUG:teuthology.nuke:shortname: smithi181 2024-05-09T03:14:53.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T03:14:53.834 INFO:teuthology.orchestra.console.smithi072:Power off 2024-05-09T03:14:53.834 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T03:14:53.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.131556', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWZvl6DWaFvDjAubG4RwJqIdDPPiqZ7+JXnpDETpJZ+YnyXfJCg9qp3bhHt2WlZCt7MkzgHzvZKEwQWtRqRi5A='} 2024-05-09T03:14:53.850 DEBUG:teuthology.nuke:shortname: smithi195 2024-05-09T03:14:53.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T03:14:53.859 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-05-09T03:14:53.859 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:53.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2024-05-09_02:35:04-rados:thrash-wip-sjust-balanced-read-testing-2024-05-08-distro-default-smithi/7699454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:44:16.132933', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY4lZiaVWRxSjv4ZaODKGlHFKK4EnLuUQ37HMQtVXu99F/L6Kjnnbd5S8OdOHhcGHfZDVr05ctdvD6JJ63RnYw='} 2024-05-09T03:14:53.894 INFO:teuthology.orchestra.console.smithi096:Power off 2024-05-09T03:14:53.895 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T03:14:53.909 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-05-09T03:14:53.919 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-05-09T03:14:53.919 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:53.938 INFO:teuthology.orchestra.console.smithi181:Power off 2024-05-09T03:14:53.939 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T03:14:53.950 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-05-09T03:14:53.963 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-05-09T03:14:53.963 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:53.977 INFO:teuthology.orchestra.console.smithi195:Power off 2024-05-09T03:14:53.978 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T03:14:53.989 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-05-09T03:14:54.015 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-05-09T03:14:54.015 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:54.041 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-05-09T03:14:57.910 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:57.951 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:57.991 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:58.021 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-05-09T03:14:58.042 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:14:58.062 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-05-09T03:14:58.102 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-05-09T03:14:58.153 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-05-09T03:15:02.022 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:02.063 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:02.103 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:02.132 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-05-09T03:15:02.154 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:02.175 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-05-09T03:15:02.213 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-05-09T03:15:02.263 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-05-09T03:15:02.315 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-05-09T03:15:02.365 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-05-09T03:15:06.136 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:06.177 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T03:15:06.247 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-05-09T03:15:06.286 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-05-09T03:15:06.348 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-05-09T03:15:06.387 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-05-09T03:15:06.492 INFO:teuthology.run:Summary data: description: rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} duration: 1197.6974120140076 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_sjust@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=325d9143fe8947ab8a2571e7cd325337 status: fail success: false 2024-05-09T03:15:06.493 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T03:15:06.576 INFO:teuthology.run:FAIL